{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "04_Linear_Regression",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "bOChJSNXtC9g",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Linear Regression"
      ]
    },
    {
      "metadata": {
        "id": "OLIxEDq6VhvZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/logo.png\" width=150>\n",
        "\n",
        "In this lesson we will learn about linear regression. We will first understand the basic math behind it and then implement it in Python. We will also look at ways of interpreting the linear model.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "VoMq0eFRvugb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Overview"
      ]
    },
    {
      "metadata": {
        "id": "-qHciBsX93ej",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/linear.png\" width=250>\n",
        "\n",
        "$\\hat{y} = XW$\n",
        "\n",
        "*where*:\n",
        "* $\\hat{y}$ = prediction | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n",
        "* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n",
        "* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ "
      ]
    },
    {
      "metadata": {
        "id": "QAgr7Grv9pb6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* **Objective:**  Use inputs $X$ to predict the output $\\hat{y}$ using a linear model. The model will be a line of best fit that minimizes the distance between the predicted and target outcomes. Training data $(X, y)$ is used to train the model and learn the weights $W$ using stochastic gradient descent (SGD).\n",
        "* **Advantages:**\n",
        "  * Computationally simple.\n",
        "  * Highly interpretable.\n",
        "  * Can account for continuous and categorical features.\n",
        "* **Disadvantages:**\n",
        "  * The model will perform well only when the data is linearly separable (for classification).\n",
        "  * Usually not used for classification and only for regression.\n",
        "* **Miscellaneous:** You can also use linear regression for binary classification tasks where if the predicted continuous value is above a threshold, it belongs to a certain class. But we will cover better techniques for classification in future lessons and will focus on linear regression for continuos regression tasks only.\n"
      ]
    },
    {
      "metadata": {
        "id": "xP7XD24-09Io",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Training"
      ]
    },
    {
      "metadata": {
        "id": "476yPgTM1BKJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "*Steps*: \n",
        "1. Randomly initialize the model's weights $W$.\n",
        "2. Feed inputs $X$ into the model to receive the predictions $\\hat{y}$.\n",
        "3. Compare the predictions $\\hat{y}$ with the actual target values $y$ using the objective (cost) function to determine loss $J$. A common objective function for linear regression is mean squarred error (MSE). This function calculates the difference between the predicted and target values and squares it (note that the $\\frac{1}{2}$ is just used to simplify the derivative operation):\n",
        "  * $MSE = J(\\theta) = \\frac{1}{2}\\sum_{i}(\\hat{y}_i - y_i)^2$\n",
        "4. Calculate the gradient of loss $J(\\theta)$ w.r.t to the model weights:\n",
        "  * $J(\\theta) = \\frac{1}{2}\\sum_{i}(\\hat{y}_i - y_i)^2 = \\frac{1}{2}\\sum_{i}(X_iW - y_i)^2 $\n",
        "  * $\\frac{\\partial{J}}{\\partial{W}} = X(\\hat{y} - y)$\n",
        "4. Apply backpropagation to update the weights $W$ using a learning rate $\\alpha$ and an optimization technique (e.g. stochastic gradient descent). The simplified intuition is that the gradient tells you the direction for how to increase something so subtracting it will help you go the other way since we want to decrease loss $J(\\theta)$:\n",
        "  * $W = W- \\alpha\\frac{\\partial{J}}{\\partial{W}}$\n",
        "5. Repeat steps 2 - 5 until the model performs well."
      ]
    },
    {
      "metadata": {
        "id": "jvJKjkMeJP4Q",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Data"
      ]
    },
    {
      "metadata": {
        "id": "RuPl9qlSJTIY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We're going to create some simple dummy data to apply linear regression on."
      ]
    },
    {
      "metadata": {
        "id": "HRXD7LqVJZ43",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from argparse import Namespace\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NFsKg-Z6IWqG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Arguments\n",
        "args = Namespace(\n",
        "    seed=1234,\n",
        "    data_file=\"sample_data.csv\",\n",
        "    num_samples=100,\n",
        "    train_size=0.75,\n",
        "    test_size=0.25,\n",
        "    num_epochs=100,\n",
        ")\n",
        "\n",
        "# Set seed for reproducability\n",
        "np.random.seed(args.seed)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NWux2lcoIWss",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Generate synthetic data\n",
        "def generate_data(num_samples):\n",
        "    X = np.array(range(num_samples))\n",
        "    random_noise = np.random.uniform(-10,10,size=num_samples)\n",
        "    y = 3.65*X + 10 + random_noise # add some noise\n",
        "    return X, y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2mb2SjSQIWvF",
        "colab_type": "code",
        "outputId": "a56ff9a8-e3ad-40d5-99c5-af8add180901",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "cell_type": "code",
      "source": [
        "# Generate random (linear) data\n",
        "X, y = generate_data(args.num_samples)\n",
        "data = np.vstack([X, y]).T\n",
        "df = pd.DataFrame(data, columns=['X', 'y'])\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>X</th>\n",
              "      <th>y</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.0</td>\n",
              "      <td>3.830389</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.0</td>\n",
              "      <td>16.092175</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.0</td>\n",
              "      <td>16.054555</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3.0</td>\n",
              "      <td>26.657172</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4.0</td>\n",
              "      <td>30.199516</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     X          y\n",
              "0  0.0   3.830389\n",
              "1  1.0  16.092175\n",
              "2  2.0  16.054555\n",
              "3  3.0  26.657172\n",
              "4  4.0  30.199516"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "metadata": {
        "id": "6LwVVOkiLfBN",
        "colab_type": "code",
        "outputId": "5116e8fc-cf2c-40b0-8e48-a5a94f0bf8e1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 362
        }
      },
      "cell_type": "code",
      "source": [
        "# Scatter plot\n",
        "plt.title(\"Generated data\")\n",
        "plt.scatter(x=df[\"X\"], y=df[\"y\"])\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeQAAAFZCAYAAABJ+lxSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Wt8VOW5N/7fnIdJJiQTZtxyqoKg\nKIcEI1uEiImJSbTWKMedTdWaUt3QViqKiKhYbRFEbC3sB5WDVHc/pMZD2Z+i4aGAG/tAqgQ57e5N\nwb8WQiCZZEgmmcwhw/q/oDMkkzVrTSYryRx+3zeamZU1K7fBi/u+r/u6VIIgCCAiIqIBpR7oByAi\nIiIGZCIiopjAgExERBQDGJCJiIhiAAMyERFRDGBAJiIiigEMyEQKEwQBv/3tb/G9730PJSUlKCws\nxIIFC3D8+PGBfjQAwP79+3Hu3Lkef9+NN96Is2fPKnKd3W7Hn/70px4/A1EiY0AmUtjrr7+OP/7x\nj9i0aRM++eQTfPrpp7jzzjvxgx/8AE1NTQP9eHjnnXeiCshKqq6uxp49ewb0GYhiDQMykYIuXryI\nbdu2YfXq1bDZbAAAjUaDefPmYe/evbBYLACA8+fP47HHHkNRURGKiorw2WefAQDOnj2L6dOn47e/\n/S3uvfde5ObmYufOnQAuz7zXr1+PoqIi5OXl4eWXX4bf7wcAfP/738frr7+OkpIS1NTUwG63o7y8\nHMXFxcjPz8fWrVsBAL/61a9w8OBBPPXUU9i5cye8Xi9efvllFBUVIT8/Hxs3bgz+LJ999hkKCwtR\nUlKCTZs2hf2Zpa7bsGEDioqKUFBQgEcffRQtLS04ceIEfv7zn6Oqqgo/+9nPAADvv/8+SkpKcNdd\nd+Ff//VfUVtbq8R/DqL4IhCRYvbt2ycUFRXJXvfggw8Kr7/+uiAIgvDNN98IU6ZMEZqamoQzZ84I\nN954o/Duu+8KgiAIO3fuFAoLCwVBEISPPvpIuOeee4SWlhbB5/MJP/rRj4LXzZ8/X3jkkUcEv98v\nCIIg/PznPxeef/55QRAE4e9//7tw0003CefOnRMEQRDy8vKEL774QhAEQVi/fr3w0EMPCR6PR2hr\naxNKS0uFPXv2CB0dHcK0adOE/fv3C4IgCJs3bxbGjh0rnDlzpsvPIXXdsWPHhKlTpwpOp1Pw+/3C\nww8/LGzYsEEQBEF44403hOXLlwuCIAh2u10YP368UFdXJwiCICxbtiz4HlEy4QyZSEHNzc3BWTAA\ntLS0oLi4GMXFxbj99tvx9ttvw+Vyobq6Gg8//DAA4Dvf+Q5uvvnm4Cy5o6MDDzzwAADgpptuCi4v\n7927FzNnzoTZbIZWq8Xs2bOxa9eu4GfNmDEDavXlP9IrVqzAc889BwAYMWIErFar6L7u3r17UVZW\nBr1eD5PJhPvuuw+7du3CN998A6/Xi+nTpwMA7r//ftGfV+q68ePHY9++fUhNTYVarUZ2djbOnDnT\n7R6ZmZk4dOgQ/umf/gkAkJOTI3odUaLTDvQDECUSi8WC+vr64NdpaWn49NNPAQDPPvss3G43nE4n\nBEHAvHnzgte5XC7ceuutAC4vcZtMJgCAWq3GpUuXAABOpxObN29GRUUFAMDv93cJ/oMHDw7++7Fj\nx/Daa6+hrq4OarUaDQ0Nwft05nQ6sWrVKqxbtw4A4PV6MXHiRDQ3NyM1NVX03p1JXdfe3o5Vq1ah\nuro6eO0dd9zR7R5+vx9vvPEG9uzZA7/fj7a2Nlx77bWin0eUyBiQiRSUlZWFxsZG/Pd//zduvPFG\n0WsyMzOh0WjwwQcfICUlpct7UtnJNpsN+fn5mD9/vuxzPPXUU3jooYfwL//yL1CpVMjNzQ17z0ce\neQR5eXldXj99+jRaW1uDX4dLRhs8eHDY67Zt24ZvvvkGH374IVJSUvD666/jwoUL3e6xc+dO7Nmz\nB++99x4sFgt+//vf4z//8z9lf0aiRMMlayIFpaamYuHChVi6dCm+/fZbAMClS5fwxz/+EZ988glG\njhwJrVaLGTNmYPv27QAuzySfeeYZ1NXVSd77zjvvxB/+8Ae0t7cDALZv346PPvpI9NrGxkaMHz8e\nKpUKH330Edrb2+FyuQAAWq0WTqczeM/3338ffr8fgiDg3//93/Ff//VfGDlyJDQaTXB2++GHH0Kl\nUnX7HKnrGhsbMWrUKKSkpKC2thafffaZ6DM0NjZi2LBhsFgscDgc+OSTT9DW1hbhiBMlDgZkIoUt\nWLAADz30EH7605+iuLgYBQUF+PDDD/HGG2/gvvvuAwCsXLkSX3zxBYqLi3H//fdjxIgRuPrqqyXv\nW1BQgLy8PNx///0oLi7Gnj17gnu3oR5//HEsWrQI9957L1wuF+bOnYvnnnsOf//731FUVIQnnngC\nW7duRVlZGYYOHYp77rkHxcXFOH36NG6++WbodDq89NJLWL58OUpKSqBSqYLL6J1JXTdv3jx88cUX\nKCoqwurVq7Fs2TIcOHAA77zzDqZNm4aDBw9i5syZ+O53v4uLFy+isLAQS5YsweLFi3H+/Hm88sor\nvfwvQRRfVILAfshEREQDjTNkIiKiGMCATEREFAMYkImIiGIAAzIREVEMYEAmIiKKAQNaGKShwano\n/TIyTHA4XIreMxlxHJXBcVQGx1EZHEdl9HYcrVZz2PcSaoas1WoG+hESAsdRGRxHZXAclcFxVEZf\njmNCBWQiIqJ4xYBMREQUAxiQiYiIYgADMhERUQxgQCYiIooBDMhEREQxgAGZiIgoBjAgExERifD4\n/Kh3uODx+fvl8wa0UhcREVGs8V+6hIo9p3D4ZAOaWjywpBmQPdaKufnX9enncoZMRETUScWeU9j9\n5Vk0tnggAGhs8WD3l2dRsedUn34uAzIREdE/eHx+HD7ZIPre4ZN2uL0dffbZDMhERJSUxPaIm1s9\naGrxiF7vcLrhCPOeEriHTERESUVqj3hwqgGWNAMaRQJvhtmIjDQDnM3tffJcnCETEVFSkdojNug0\nyB5rFf2+7LFDYNT33TyWAZmIiJKG3B6xx+fH3PzrUJAzHJlpRqhVQGaaEQU5w/s8y5pL1kRElDTk\n9oibWz2wZZhQVjAWM2eMRnOrB4NTDTDo+r6fNGfIRESUNAJ7xGIyzEYMTr3ynkGngS3D1C/BGGBA\nJiKiJOHx+dHc6sHE64aIvp89dki/BV8xXLImIqKEFppVnWHWY4QtFS63Dw6nBxlmI7LHDunzPWI5\nDMhERBSXAjNeuT3eQFZ1QJPTiyanF3nZQ1E0ZWTw+z0+PxqbXf22ZxyKAZmIiOKK1DlijbrrTqxU\nVvXR002Ykz8GWo0Kv9t9MqL79SUGZCIiiiuhM97AOWIAKCsYG3zd4/Pj69pm2azq3YfORnS/vsaA\nTEREcUPuHPHMGaOh1aiCM+jGFg/UKkAQul+fYTZikEEre7/+Wr5mQCYiorgRyTni0BnvJZFgDFzO\nqm73dER0Lrk/8NgTERHFDalzxOmpBrS6fWFnvGoVoAqpvNWTc8l9jTNkIiKKG4Fa051nwAEuTwde\n3nYo7PcKAvDkvCyMGjY4uAytUSPs/fr7XLJsQG5vb8eyZcvQ2NgIj8eDhQsXoqqqCidOnEB6ejoA\noLy8HHfccQd27NiBbdu2Qa1WY86cOZg9e3af/wBERJRcAueFD5+0w+F0Q6/TwO31w+31S36fJc3Y\nJRiHu99AnUuWDch79+7F+PHjsWDBAtTW1uKRRx5BdnY2nnjiCeTl5QWvc7lc2LBhAyorK6HT6TBr\n1iwUFhYGgzYREZESNGp1sNZ0g8OFX1celQ3GQPgZb+f79Wft6lCyAfnuu+8O/ntdXR2uuuoq0euO\nHDmCCRMmwGw2AwAmT56Mmpoa5OfnK/SoRESUrMSKgBh0Guh1mrBJWcDlPWNLhDPeQO3qgRLxHvK8\nefNw/vx5bNy4Ee+88w7ee+89bN26FZmZmXjuuedgt9thsViC11ssFjQ0iG+sExERRUKuCEggKatR\nJChbzAYsnjMJ1vRBA1qjOlIRB+Tt27fjr3/9K5566iksX74c6enpGDduHN566y2sX78e2dnZXa4X\nxA59hcjIMEGrVXaQrFazovdLVhxHZXAclcFxVEY8jaPb2wFHiwcff/61aNEO0yA9FpROAABMmzQM\nO/Z/3e0e07OGIfvGqxV/tr4aR9mAfPz4cWRmZuLqq6/GuHHj4Pf7MXbsWGRmZgIA8vPzsXLlShQV\nFcFutwe/r76+HllZWZL3djhcvXz8rqxWMxoanIreMxlxHJXBcVQGx1EZ8TKOnWfEgaIeYv585BxK\npoyAQafBvVNHwtXu7ZaUde/UkYr/zL0dR6lgLnsO+csvv8SWLVsAAHa7HS6XC88//zzOnDkDAKiu\nrsaYMWMwadIkHDt2DC0tLWhra0NNTQ1ycnKifmgiIko+gbKYgSXocEU9AkU7gCtJWS8v+Gf88ke3\n4uUF/4yygrH9WodaCbIz5Hnz5uHZZ59FWVkZ3G43nn/+eZhMJixevBiDBg2CyWTCqlWrYDQasWTJ\nEpSXl0OlUmHRokXBBC8iIiI5UmUxQ4kV7RjopKzekg3IRqMRr732WrfXP/jgg26vFRcXo7i4WJkn\nIyKipCJVFjNUfxft6A+s1EVERDFBKmNarQIERH6EKR4xIBMRUUyQKos5I2soiqaMHLCiHf2BAZmI\niBQXWshDrLCHGKkylvGWpNVTDMhERKSY0EIeGWY9Ugbp4XL7uhT2KM29Fq0uX7cAHStlLAcCAzIR\nESkmcGwpoMnpRZPTG/w6UNjj86N18Hj9YQN0vGdMR4MBmYiIFNGTY0uBZhDhAnQyLFGHSq6floiI\n+oTH58fXtc0RH1sK5fb6IeBKgK7Yc0rZB4wDnCETEVHUxEpdRtDKQNbhk3bMnDE6afaPAc6QiYio\nFyItddlTnUtjJgvOkImIKCpSe8aBQh4ZqQakDNLB5fbB4fRAr9ME94+liJXGTHQMyEREFBWpUpeC\nADw5Lwujhg3ucg451aTHx/u/Dp4zDhegE7E0phwGZCIiiopUqUtLmjEYjIGujR86nzMODdAZCVwa\nUw4DMhERyRKrtCVV6lJuhhsuQCdTIZBQDMhERBRWaOWt0HPCUqUueyIZC4GEYkAmIkpyUnWmQytv\nBc4JA5dntslc6lJpDMhERElKbvYrlUUdek6YM9zeY0AmIkpSUrPfmTNGS1beCpwTZhBWDgMyEVES\nkpr9fn60TrbyVnqqAd6OS/D4/FyiVggDMhFREpI6Q+z2+oNng8NV3nJ5OvDC5r8kdTMIpXH0iIiS\nUOAMcaTUKkClAoz6y7NhNoNQHgMyEVESCpwhjpQgAI/PnIgUo/jC6uGTdnh88iUxKTwGZCKiJOPx\n+VHvcKE0dxQKcoYjM80ItQqwmA3BGXAoS5oRlsFG2SQvih73kImIkkS4Y04vlt+CVpcPg1MN+OCz\n02Erb1nTB4UtlZmMzSCUxoBMRJQk5Ip8AJCsvKVRq6MulUnyGJCJiJKA29sRUZEPucpbSpXKpO4Y\nkImIkoCjJfwxJ7EiH+Eqb7FUZt9hUhcRURLISAt/zCma/d9AwGYwVg4DMhFRgvP4/HC0eDDxuiGi\n73P/NzbILlm3t7dj2bJlaGxshMfjwcKFC3HDDTdg6dKl8Pv9sFqtePXVV6HX67Fjxw5s27YNarUa\nc+bMwezZs/vjZyAiIhFdsqqdHmSk6jHClgqX2weH08P93xgjG5D37t2L8ePHY8GCBaitrcUjjzyC\nyZMno6ysDCUlJVi3bh0qKytRWlqKDRs2oLKyEjqdDrNmzUJhYSHS09P74+cgIqIQoVnVTU4vmpxe\n5GUPRdGUkdz/jTGyS9Z33303FixYAACoq6vDVVddherqatx5550AgLy8PBw4cABHjhzBhAkTYDab\nYTQaMXnyZNTU1PTt0xMRkSip5hFHTzcxGMegiLOs582bh/Pnz2Pjxo34wQ9+AL1eDwDIzMxEQ0MD\n7HY7LBZL8HqLxYKGBvFfhoCMDBO0WmV/IaxWs6L3S1YcR2VwHJXBcewZt7cD5751oMkZPqtao9fB\nOiSln58sMfTV72PEAXn79u3461//iqeeegpCp15cglhfLonXO3M4XJF+fESsVjMaGpyK3jMZcRyV\nwXFUBscxcp33jIOtE0WuyzAb4ff6OK5R6O3vo1Qwl12yPn78OOrq6gAA48aNg9/vR0pKCtxuNwDg\nwoULsNlssNlssNvtwe+rr6+HzWaL+qGJiJJVoNZ0uGYN4d4P7BkHSluGa53IrOrYJDtD/vLLL1Fb\nW4tnn30WdrsdLpcLubm5qKqqwn333Yddu3YhNzcXkyZNwooVK9DS0gKNRoOamhosX768P34GIqKE\nEK7WdKBspdT7HX4h7J6xWnX5n8yqjm2yAXnevHl49tlnUVZWBrfbjeeffx7jx4/H008/jYqKCgwd\nOhSlpaXQ6XRYsmQJysvLoVKpsGjRIpjN3PchIoqUXK1pqfcLbh4ethKXIAAvPXYbMlN0nBnHMNmA\nbDQa8dprr3V7fevWrd1eKy4uRnFxsTJPRkSURKSyog+ftOPe266RfT9cJyZLmhHXfycDzuZ2RZ+Z\nlMVKXUREMaC5VbrW9Nn6Vsn32z0dyB5rFX0/e+wQGPVsXRDr+F+IiCgGDE41hJ3hpqcaYDBoJN/3\ndlxCae4oAOzEFK8YkImIYoBBpwnba9jl6cAvth2CQS++qOnydOCFzX8JJnm9WH4LWl0+Fv+IMwzI\nREQxIrTXsF6ngdvrh9t7+XiT23sJAGDUa+D1+bu9H5oERvGFe8hERDEi0Gv45QX/jJU/uAUpRvE5\nk8mgxfIHbw77/uGT9rBnmCl2MSATEcUYg04DvU4TNonrYqsHHo9fMsmruVX8PYpdDMhERDEokOQl\nJsNsxHBbquT7g1PF36PYxYBMRBSDAkleYrLHDoHZpJd8n8lc8YdJXUREMSo0ySv0GJPc+xRfGJCJ\niAaYx+dHc6un2zGlQJLXzBmjo3qf4gsDMhHRAJFrJhFg0GlgyzCFvY/c+xQfGJCJiPpQuNkvIN9M\ngpILAzIRUS+EBtzA16kmPT7e/3XY2a9cM4mZM0Zz+TnJMCATEUUhdLk5w6xHyiA9XG4fmlo8MOjV\nwcpaQNfZ78wZo/F1bbPsOWIuQycXBmQioiiELjc3Ob1ocnqDX3cOxp19frQOh082oLHFA7Xqcq/i\nUDxHnJwYkImIekhquVlO59rTl0SCMcBzxMmKAZmIqBOpJKwAqd7F0VCrAAGAheeIkxoDMhERIj+C\nBEj3Lo6GIABPzsvCqGGDOTNOYiydSUSEK3vCjS0eCLiShFWx51S3a6XKWoYy6jVQqwCL2QCjXjzY\nWtKMDMbEGTIRUTRHkELLVqanGpAySAeX2weH0xMsY1maey1aXT4MTjXgg89Od0kEC+CeMQEMyERE\nknvC4Y4ghStbKbYHbTLoALD2NEljQCaipCe1Jyx3BCm0bKVUGUvWniYp3EMmoqQn1+pQ6aAZCNoM\nxtQZZ8hEROByMg08BmQiIkS+nBzJOWWiaDAgExF1Em4PuCfnlImiwYBMRCQhMCOu+uIM9tbUBl9n\nq0RSWkQBec2aNTh06BA6Ojrw6KOPYs+ePThx4gTS09MBAOXl5bjjjjuwY8cObNu2DWq1GnPmzMHs\n2bP79OGJiHor3BJ05xlxoBGEGLZKJKXIBuSDBw/ib3/7GyoqKuBwOHD//ffj1ltvxRNPPIG8vLzg\ndS6XCxs2bEBlZSV0Oh1mzZqFwsLCYNAmIoolckvQod2cwjWCYKtEUopsQL7lllswceJEAEBaWhra\n29vh9/u7XXfkyBFMmDABZrMZADB58mTU1NQgPz9f4UcmIuq90IAb2q840m5ObJVISpHNRNBoNDCZ\nLv/Nr7KyErfffjs0Gg3ee+89PPjgg/jZz36GpqYm2O12WCyW4PdZLBY0NETXnoyIqC/JlcpsuNge\ncTcnlr0kpUSc1LV7925UVlZiy5YtOH78ONLT0zFu3Di89dZbWL9+PbKzs7tcL4h13Q6RkWGCVqvs\nL7LValb0fsmK46gMjqMylB7HOnsbmpzhS2VmZKTAmjEI9Y72bu+r1QAEYEj6INw6/mo8cu9N0Gji\nI8uav4/K6KtxjCgg79+/Hxs3bsSmTZtgNpsxderU4Hv5+flYuXIlioqKYLfbg6/X19cjKytL8r4O\nhyvKxxZntZrR0OBU9J7JiOOoDI6jMvpiHP0+Pyzm8KUytcIlTBydKdoIYsakoSiaMjKYBNbU1Kbo\ns/UV/j4qo7fjKBXMZf9a53Q6sWbNGrz55pvBBK2f/OQnOHPmDACguroaY8aMwaRJk3Ds2DG0tLSg\nra0NNTU1yMnJifqhiYiU4PH5Ue9wweO7kvsSSanMufnXoSBnODLTjFCrgMw0IwpyhqOscCzLXlKf\nkJ0h79y5Ew6HA4sXLw6+9sADD2Dx4sUYNGgQTCYTVq1aBaPRiCVLlqC8vBwqlQqLFi0KJngREfU3\nuSxquVKZbARB/U0lRLLZ20eUXj7hkowyOI7K4DgqI9px/N3uk6JLznnZXZeck6UUJn8fldGXS9as\n1EVECUcqi/qzr85h3+FzXWbMPENMsSA+UgOJiMIQ2yNubvWEPbZ0SQAEXDl3XLHnVD89KZE0zpCJ\nKG50Xl7WalRh94gHpxpgSRPPog7F0pcUKxiQiSjmiSVomYw6nKlvDV4TmPH6/ZdQNGUkJl43pEsz\niHBY+pJiBQMyEcU8sTKX4Wa/gT3iDLMeI2ypcLl9aGrxQKUSr0fN0pcUK7iHTEQDJnT/V2w/2O3t\niLiuNHBlj7jJ6cWZ+lZMHJ2JVY/eihnZw0SvZ+lLihWcIRNRvwtdgs4w65EySB+czXbeD3a0hE/Q\nisTR002Ykz8GZQVjoFGrwp47JhpoDMhE1G8CSVlVX5zpsr/b5PSiyekNft2589KjMydFnKAlpvMe\nMQt9UCxjQCaiPtd5RtzY4oFaFdn3HT55uT5+9liraJGPy3vEHWhqcUe8R2zQaZjARTGJAZmI+lxo\nUpZY4BTT1OLG/37rQGnuKADiZS47/ILorDuAe8QULxiQiahPSVXNkqNSAc+9+f9gMV/eU36x/Ba0\nunxdlps1avxjOZp7xBTfGJCJqE9JVc2SE5hJd95TLisYK3otm0FQvOOxJyLqU4GqWWLUqsuzYIvZ\ngBG2VGSmGaD6x+tiDp+0dzkSJSawR8xgTPGGM2Qi6jOBrOpwVbNmZHXvvPR1bTPWbv9K9H6sqkWJ\njAGZiBQTCMCpJj0+3v91l3PGgapZDqeny/6uRn1loc6g02DUsMFhjzmxqhYlMgZkIuq10EIfBr0a\nbu+l4PuBc8ahvYjFGHSasMecmDFNiYx7yETUa4FjTY0tHghAl2Dc2dHTTRElW83Nvw4FOcORmWaE\nWgVkphlRkDOcGdOU0DhDJqJe6cmxpkj3gDtnTGv0Ovi9Ps6MKeFxhkxEvdKTY0093QM26DS4ekgK\ngzElBQZkIuoVqWNNobgHTBQeAzIR9UogCUuMUa/hHjBRhLiHTERRCxxzCldrujT32m6lLolIHAMy\nEfVY6DGnQP9isVrTJoNugJ+WKD4wIBNRj4V2b4qk1jQRSeMeMhH1iNQxp0hqTROROAZkIuoRqWNO\ngXPGRNRzDMhE1CNSx5xYa5ooegzIRBSWx+dHvcPVZRla6pgTzxkTRS+ipK41a9bg0KFD6OjowKOP\nPooJEyZg6dKl8Pv9sFqtePXVV6HX67Fjxw5s27YNarUac+bMwezZs/v6+YmoD4TLog50ZwqcJw49\n5sRzxkTRkw3IBw8exN/+9jdUVFTA4XDg/vvvx9SpU1FWVoaSkhKsW7cOlZWVKC0txYYNG1BZWQmd\nTodZs2ahsLAQ6enp/fFzEFEvBc4UD0414IPPTktmUXeuNR34Hs6MiXpHNiDfcsstmDhxIgAgLS0N\n7e3tqK6uxosvvggAyMvLw5YtW3DttddiwoQJMJvNAIDJkyejpqYG+fn5ffj4RNRbobPhDLMeLo94\npvThk3bMnDE6GHwNOo1sowgiioxsQNZoNDCZLv+Bq6ysxO23347PP/8cer0eAJCZmYmGhgbY7XZY\nLJbg91ksFjQ0SHeAycgwQatV9m/VVqtZ0fslK46jMuJhHN/++FiX2XCT0xv2WofTDY1eB+uQlP54\ntKB4GMd4wHFURl+NY8SFQXbv3o3Kykps2bIFd911V/B1QRBErw/3emcOhyvSj4+I1WpGQ4NT0Xsm\nI46jMuJhHD0+P/58pDbi69NTDThf39Kv7RDjYRzjAcdRGb0dR6lgHlFA3r9/PzZu3IhNmzbBbDbD\nZDLB7XbDaDTiwoULsNlssNlssNvtwe+pr69HVlZW1A9NRH2vJ60TAcDl6cALm//SLcmLiHpP9k+S\n0+nEmjVr8OabbwYTtG677TZUVVUBAHbt2oXc3FxMmjQJx44dQ0tLC9ra2lBTU4OcnJy+fXoi6hWp\nM8VGvQaZaQaoVZf/HQDcXj8EXEnyqthzqh+fliixyc6Qd+7cCYfDgcWLFwdfe+WVV7BixQpUVFRg\n6NChKC0thU6nw5IlS1BeXg6VSoVFixYFE7yIKDYFzhR33kMOmD7xasycMRoNDhd+XXkUbm/3RK/Q\nJC8iip5sQJ47dy7mzp3b7fWtW7d2e624uBjFxcXKPBkRKarzsabOAVTqTLFGrYZep5EtlclMa6Le\nY7cnogQnV+RD7kxxYFm7USQos1QmkXKYjUGU4AKtEhtbPJL7v4EzxaHLzyyVSdQ/GJCJEphUq8Sa\n/23A2YbWiNolzs2/DgU5w5GZZoRaBWSmGVGQM5ylMokUxCVrogQhtkcsdaypyemJ+AgTS2US9T0G\nZKI4F26PuDT3Wnh9/rD7vwC6LGEDl+tUS2GpTKK+w4BMFOcCe8QBgQD7+dE6eLx+GPSR7UzxCBPR\nwOIeMlEck9ojDhTxcHsvAbhc3EMlca/AESYiGhgMyERxrCelL00GLZ596GZkhqnMxSNMRAOLAZko\njkmVvgx1sdWDVKOOR5iIYhQDMlGc8Pj8qHe4uhxTkjojHCowA+YRJqLYxKQuohgnV2krtPSlXqcR\nrTvdeQbMI0xEsYcBmShGBc6ToUzIAAAZQ0lEQVQVV31xBntrrvQsDj2mFHpGONWkx8f7vxatTd0Z\njzARxRYGZKIYEQjAVwJqAxpbPFCHSY0OPabUOcByBkwUfxiQiQZY6JK0Qa8OHlUCgEuC+Pc1tbjx\ndW0zRg0bLBpwOQMmii8MyEQDLLSwR+dgLEWlAtZu/yqi0pdEFPv4p5doAEkV9pBzSYBk9yYiii8M\nyEQDqCeFPQJ7yVJ7ypF0biKi2MSATDSAelLYY0bWUDw1LwtCmD1llr4kim8MyEQDSKqwh1Gv6VK4\no6xwLEYNGxw2gLP0JVF8Y1IXkQLEehFHKrSwR+DccGnutWh1+brcU6MGssdauySBBbD0JVF8Y0Am\n6gWpKlqRCi3s0TkAmwy6bteHC+AsfUkU3xiQiXohXC9iAHj8X27u0b0iPTcsFcCJKH4xIBNFSerI\n0uGTdri9HX36+Sz8QZRYmNRFFCWpI0sOpxuOCI8zEREBDMhEERFrfSh1ZCnDbESGzHEmsXsSUfLi\nkjWRBKmkrcCRpXAZz0a9Fs5/fN05C1urUUm2UySi5MSATCRBKmmrrGCsbMazWEA3GXU4U98a9p5E\nlJwiCsgnT57EwoUL8fDDD2P+/PlYtmwZTpw4gfT0dABAeXk57rjjDuzYsQPbtm2DWq3GnDlzMHv2\n7D59eKK+JJe0FWh9KJXxLBbQG8PsLYe2UySi5CIbkF0uF1566SVMnTq1y+tPPPEE8vLyuly3YcMG\nVFZWQqfTYdasWSgsLAwGbaJ4I5e01dzqCWY5i2U8u70dPWocEXpPIkoushtWer0eb7/9Nmw2m+R1\nR44cwYQJE2A2m2E0GjF58mTU1NQo9qBE/U0qaSs91QBvx6WwCVkenx//+60j4sYRAEtfEiU72Rmy\nVquFVtv9svfeew9bt25FZmYmnnvuOdjtdlgsluD7FosFDQ3RtZUjigVSSVsuTwde2PyXbglZnfeM\nG1s8UKsQthlEKJa+JEpuUSV13XfffUhPT8e4cePw1ltvYf369cjOzu5yjRDB/4UyMkzQapX9H5DV\nalb0fsmK43jZj+dkwzRIj4PH62C/2A6DXoN2jx9u7+WZcSAhyzRIjwWlE/D2x8e6BPBLYf4YjBqa\nhtZ2H+wX2zEkfRBuHX81Hrn3Jmg0zLIWw99HZXAcldFX4xhVQO68n5yfn4+VK1eiqKgIdrs9+Hp9\nfT2ysrIk7+NwuKL5+LCsVjMaGpzyF5IkjmNXpdOuQcmUEWhwuPDryqNo93Rfpv7zkXO4M3so/nyk\nVvQeahUgALB0ysLu8AtdEsGamtr6+CeJT/x9VAbHURm9HUepYB7VX8d/8pOf4MyZMwCA6upqjBkz\nBpMmTcKxY8fQ0tKCtrY21NTUICcnJ7onJupnckU6DDoN9DqNZJLX2frWsO8LAvDk3Cy8vOCfUVYw\nFhq1OpgIxmVqIgIimCEfP34cq1evRm1tLbRaLaqqqjB//nwsXrwYgwYNgslkwqpVq2A0GrFkyRKU\nl5dDpVJh0aJFMJu5PEKxTarwR2iRjkCSl9ixpQyzEcNtqWHft6QZMWrYYAZfIgpLNiCPHz8e7777\nbrfXi4qKur1WXFyM4uJiZZ6MqB/IFf7oTK4yl9mkZ69iIooaK3VRUvL4/GhwuCIq/NGZXGUu9iom\nomgxIFNSCT2WFE64Ih1yvYg7v6/R6+D3+jgzJqKIMCBTUgldog6nc+EPsYAq14vYoNPAOiSFWa1E\nFDEGZEoaUrWpQ4Ur/EFE1Ff4fxhKSGLHmKRqUwOASgUY9Zdnw26vHwKuJHlV7DnV149MREmOM2RK\nKFLHmKSOLVnMBix8YDz+z0fHg1W4OmMnJiLqa5whU0IJ7BE3tni6zXADx5bETL7eilSjTra7ExFR\nX2FApoQh17/Y4/Njbv51KMgZjsw0I9QqIDPNiIKc4V1m0GLYiYmI+hqXrCnueHx+0SNHkfYvDnds\nSaMGC3sQ0YBhQKa4IVfmUq60ZecZbrhjSyzsQUQDhQGZ4oZcmUu50paRzHDlCn8QEfUV7iFTXIhk\nfxhA2D3i0txrJbs5hWInJiLqb5whU1yQ2h9uanHj69rmYDelzjPcVJMeH+//Gi9s/otsNyciooHE\ngExxQWp/WKUC1m7/qkuwDcxwf7f7ZMTdnIiIBhKnCBQXpM4QXxIgWlUr0mVuIqJYwIBMMS9QBrM0\nd1Rwf1gFQK0Svz4QbCM5BkVEFCu4ZE0xK9wxpxfLb8G3dU6s3f6V6PcFgm1PjkEREQ00zpBpwIk1\nggDCl8H8eP//h1HDBstW1ZJa5mahDyKKNZwh04CRKvTR4Rck939nzhgd0ZljFvogonjBgEwDRqrQ\nR8HNw2X3fyMJtiz0QUTxggGZBoRcBvS9t10ju//bk2AbrlQmEVGs4B4yDQi5Qh9n61sx8bohou+H\n7v+yqhYRJQLOkGlARFLoI8OsxwhbKlxuHxxOD/d/iSihMSCT4sK1R+xMqhHEJeHyP5ucXjQ5vcjL\nHoqiKSO5/0tECY0BmRQj1x4xVOekrKYWN1SqK8G4s6OnmzAnfwyDMRElNAZkUky4rGm//5LoDLdz\nUtbXtc2yhT6YlEVEiYwBmRQhlTX92VfnsO/wueCMuTT3WrS6fMEAbdBpgoU+WFWLiJIVAzIpQipr\nOrAMHZgxf360Dh6vv1t3pkgKfRARJaqIjj2dPHkSBQUFeO+99wAAdXV1+P73v4+ysjI8/vjj8Hq9\nAIAdO3Zg5syZmD17Nt5///2+e2qKOYGs6Ui4vX7R7kxz868LNo9Qq4DMNCMKcoYzq5qIkoLsDNnl\ncuGll17C1KlTg6+98cYbKCsrQ0lJCdatW4fKykqUlpZiw4YNqKyshE6nw6xZs1BYWIj09PQ+/QEo\nNkjNcOUESmEadBpW1SKipCU7Q9br9Xj77bdhs9mCr1VXV+POO+8EAOTl5eHAgQM4cuQIJkyYALPZ\nDKPRiMmTJ6OmpqbvnpxiTucZrlR7xFChrRBZ6IOIkpHsDFmr1UKr7XpZe3s79Ho9ACAzMxMNDQ2w\n2+2wWCzBaywWCxoaxJN8KDGFlrKs+uIM9tbUyn4fk7aIiBRI6hIEkYOjEq93lpFhglar7CzIajUr\ner9k1dtxHA7ghtFWmFMMOHi8DvaL7TDoNWj3+LtdO23SUAwfmphbG/x9VAbHURkcR2X01ThGFZBN\nJhPcbjeMRiMuXLgAm80Gm80Gu90evKa+vh5ZWVmS93E4XNF8fFhWqxkNDU5F75mMlBzH0mnXoGTK\nCDS3epBq0uPj/V93685079SRCfnfjb+PyuA4KoPjqIzejqNUMI8qIN92222oqqrCfffdh127diE3\nNxeTJk3CihUr0NLSAo1Gg5qaGixfvjzqh6b4IVcqs3OnJSZtERGJkw3Ix48fx+rVq1FbWwutVouq\nqiqsXbsWy5YtQ0VFBYYOHYrS0lLodDosWbIE5eXlUKlUWLRoEcxmLo8ksp6WygxgK0Qiou5UQiSb\nvX1E6eUTLskoo/M4Ss1+f7f7pOgxp4Kc4SgrGNsvzxrL+PuoDI6jMjiOyoi5JWtKHOECrtzsV6pU\nZudzxUREFBkG5CQlFXCB8I0igMv7wFKlMtkMgoio5yIqnUmJJxBwG1s83cpYur0dkrNfj88vWSqT\n54qJiHqOATkJyS03n29sk539BkplimEzCCKinuOSdRKSW24GVJKtEAcZtKh3uFCaOwoAup0rZjMI\nIqKeY0BOMh6fH16fP2zATU81ABAw8bohomUvTUYtfv7OF132nV8sv6VLf2MiIuo5BuQkEZrEZdCL\n71a4PB346Wv7kJGqxwhbKlxuHxxODzLMRpiMWpypbw1eG5roRURE0eMecpIITeJyey8BAIx6DdSq\ny/8E/tGrWACanF6cqW/FxNGZ+OWPbsXzD+fA5faJ3juQ6EVERNFjQE4CUklcJoMWyx+8GSlG8cWS\no6ebMDjVgHZPh2yiFxERRY8BOQlIJXFdbPXA4/HLBlsecyIi6lsMyElALpgOt6XKBlsecyIi6lsM\nyElALpiaTfqIgu3c/OtQkDMcmWlGqFVAZpoRBTnDecyJiEgBzLJOEoGgGe7MsNz7AKBRq9k+kYio\nj7DbU5KR613s8fmh0evg9/oYbHuJv4/K4Dgqg+OojL7s9sQl6wTj8flR73CFPYYU6EUcLtgadBpc\nPSSFwZiIqJ9xyTpByLVLJCKi2MaAHOcCS9BVX5zpUuqSVbSIiOILA3Kc6jwjbmzxQK0Sv+7wSTtm\nzhjNJWgiohjHtcw41bkUJgBcCpOa19Tixte1zSxtSUQU4zhDjnFiWdFSpTBDqVTA2u1fcU+ZiCjG\nMSDHKKkkLalSmKECM2fuKRMRxTZOlWJUaHemQED93f89GexnLCawlyy1p8zlayKi2MMZcozx+Pxo\ncLjCLkl/9tU57Dt8Lmw/4xlZQ5FzvQ1rt38l+n6gWYQtw6TYMxMRUe8xIMeI0KzpcAJL0J37GXt9\n/i6lLjv8AixpBtH7sDMTEVFsYkCOEYEl6p4yGbRY/v2bYU0fFEz60qiB7LFW0fuxMxMRUWxiQI4B\nPcmaDnWx1QO9Vt0tyEbSLIKIiGIHA3IMiCRrWq0SP2scbgmanZmIiOILs6z7kFyjh8D7gwzasFnT\nFrMBPy+fghnZw0Tfl1uClmsmQUREsSGqGXJ1dTUef/xxjBkzBgAwduxY/PCHP8TSpUvh9/thtVrx\n6quvQq/XK/qw8UKu0YPY+yajTjQJa/L1Vgy3pqKsYAw0ahWXoImIElTUS9ZTpkzBG2+8Efz6mWee\nQVlZGUpKSrBu3TpUVlairKxMkYeMF5E2eghN4Gps8aCxxYMRtlS43B2iAZdL0EREiU2xPeTq6mq8\n+OKLAIC8vDxs2bIlaQJyTxo93HvbNWETuFzuDjz/cA7aPR1hA25gCZqIiBJL1AH51KlTeOyxx9Dc\n3Iwf//jHaG9vDy5RZ2ZmoqFBPms4I8MErVbZWZ7Valb0fpF4++NjXWa84Ro9OJxuOL2X0OQUT+By\nON0YlGLEqO+k9MVj9shAjGMi4jgqg+OoDI6jMvpqHKMKyNdccw1+/OMfo6SkBGfOnMGDDz4Iv/9K\n4pIghIlIIRwOVzQfH5bVakZDg1PRe8rx+Pz485Fa+QsBpKca4G73wGIOX7TD7/X1+88QaiDGMRFx\nHJXBcVQGx1EZvR1HqWAeVZb1VVddhbvvvhsqlQojR47EkCFD0NzcDLfbDQC4cOECbDZbdE8bZ3rS\n6MHl6cAvth1Cm9sn+j6LdhARJa+oAvKOHTuwefNmAEBDQwMaGxvxwAMPoKqqCgCwa9cu5ObmKveU\nMcrj88s2elCpLpe3BAC31w8BXcteqlVAZpoRBTnDmTFNRJTEolqyzs/Px5NPPok//elP8Pl8WLly\nJcaNG4enn34aFRUVGDp0KEpLS5V+1pgRemxJqtFDXvYw/LryKNze7meRxcpeEhFRcooqIKempmLj\nxo3dXt+6dWuvHygehB5bkmr00NjsDrukHa7sJRERJR+WzuwhqbrTYjPewakGdl4iIiJZLJ3ZQ1JJ\nXGIzXoNOg+yxVtHrmcRFREQBnCH3UDQzXnZeIiIiOQzIPRSY8fak1zDLXhIRkRwG5ChEO+Nl2Usi\nIgqHAbmTQHMIuRksZ7xERKQ0BmTIt0sMhzNeIiJSSlIH5EjbJYZezxkxEREpLSkDck/aJc6cMRpa\njSqqGTQREVGkkjIgh1bakmqX2Nzqwe5DZ7tcH24GTUREFK2km95JVdoKlZ5qQKvbF/b6wyft8Pi6\n16gmIiLqqaQLyD1tl/jytkOiRUCAKzNoIiKi3kq6gByotCVGrF2iFNaiJiIipSRdQJaqLT0jayhe\n/MEtSDFGtrXOWtRERKSUpEzqkqq0JdUuEbg8g7awFjURESksYQOy2Jnhzq+Fq7Ql1TzCYjZg8ZxJ\nXdorEhERKSHhArJY1a1JY4ZABeCrv9m7nSMOrbQl1Txi8vVWDLem9tNPQkREySThAnLoGePGFg/2\nHKrtco3cOWK2SyQiov6WUAHZ7e2I+IwxcKUSV+jyM5tHEBFRf0uoLGtHS+RnjAH5c8SB5hEMxkRE\n1NcSKiBnpIU/Yyx6Pc8RExFRjEiogGzUa8OeMRbDc8RERBQrEmoPGRBPyJo0JvMfWdaNTNIiIqKY\nlHABWSoha9Yd7GdMRESxKWECssfnR529DX6fHwadJpiQ1ZnYa0RERLEg7gNyl0IgTg8s5itFPzTq\nhNoiJyKiBBb3AVmsEIhU0Q8iIqJYpPgU8pe//CXmzp2LefPm4ejRo0rfvguPzx+2EMjhk3Z4fNLt\nE4mIiGKFogH5L3/5C7799ltUVFTgF7/4BX7xi18oeftumlvDFwKRK/pBREQUSxQNyAcOHEBBQQEA\nYPTo0WhubkZra6uSH9FFoDOTGBb9ICKieKJoQLbb7cjIyAh+bbFY0NAQeW3pngp0ZhLDoh9ERBRP\n+jSpSxAEyfczMkzQansXNH88JxumQXocPF4H+8V2DEkfhFvHX41H7r0JGg2zrKNltZoH+hESAsdR\nGRxHZXAcldFX46hoQLbZbLDb7cGv6+vrYbWGL2XpcLgU+dzSadegZMoIaPQ6+L0+GHQaNDW1KXLv\nZGS1mtHQ4Bzox4h7HEdlcByVwXFURm/HUSqYKzqFnDZtGqqqqgAAJ06cgM1mQ2pqqpIfEZZBp8HV\nQ1K4TE1ERHFJ0Rny5MmTcdNNN2HevHlQqVR44YUXlLw9ERFRwlJ8D/nJJ59U+pZEREQJj1lPRERE\nMYABmYiIKAYwIBMREcUABmQiIqIYwIBMREQUAxiQiYiIYoBKkKtvSURERH2OM2QiIqIYwIBMREQU\nAxiQiYiIYgADMhERUQxgQCYiIooBDMhEREQxQPFuTwPll7/8JY4cOQKVSoXly5dj4sSJA/1IcWPN\nmjU4dOgQOjo68Oijj2LChAlYunQp/H4/rFYrXn31Vej1+oF+zLjgdrvx3e9+FwsXLsTUqVM5jlHY\nsWMHNm3aBK1Wi5/+9Ke4/vrrOY491NbWhqeffhrNzc3w+XxYtGgRrFYrVq5cCQC4/vrr8eKLLw7s\nQ8a4kydPYuHChXj44Ycxf/581NXVif4e7tixA9u2bYNarcacOXMwe/bs6D9USADV1dXCj370I0EQ\nBOHUqVPCnDlzBviJ4seBAweEH/7wh4IgCEJTU5MwY8YMYdmyZcLOnTsFQRCE1157TfiP//iPgXzE\nuLJu3TrhgQceED744AOOYxSampqEu+66S3A6ncKFCxeEFStWcByj8O677wpr164VBEEQzp8/LxQV\nFQnz588Xjhw5IgiCIDzxxBPCvn37BvIRY1pbW5swf/58YcWKFcK7774rCIIg+nvY1tYm3HXXXUJL\nS4vQ3t4u3HPPPYLD4Yj6cxNiyfrAgQMoKCgAAIwePRrNzc1obW0d4KeKD7fccgt+/etfAwDS0tLQ\n3t6O6upq3HnnnQCAvLw8HDhwYCAfMW6cPn0ap06dwh133AEAHMcoHDhwAFOnTkVqaipsNhteeukl\njmMUMjIycPHiRQBAS0sL0tPTUVtbG1w55DhK0+v1ePvtt2Gz2YKvif0eHjlyBBMmTIDZbIbRaMTk\nyZNRU1MT9ecmREC22+3IyMgIfm2xWNDQ0DCATxQ/NBoNTCYTAKCyshK333472tvbg0uCmZmZHMsI\nrV69GsuWLQt+zXHsubNnz8LtduOxxx5DWVkZDhw4wHGMwj333INz586hsLAQ8+fPx9KlS5GWlhZ8\nn+MoTavVwmg0dnlN7PfQbrfDYrEEr+lt7EmYPeTOBFYD7bHdu3ejsrISW7ZswV133RV8nWMZmY8/\n/hhZWVkYMWKE6Pscx8hdvHgR69evx7lz5/Dggw92GTuOY2T+8Ic/YOjQodi8eTP+53/+B4sWLYLZ\nbA6+z3HsnXDj19txTYiAbLPZYLfbg1/X19fDarUO4BPFl/3792Pjxo3YtGkTzGYzTCYT3G43jEYj\nLly40GXZhsTt27cPZ86cwb59+3D+/Hno9XqOYxQyMzORnZ0NrVaLkSNHIiUlBRqNhuPYQzU1NZg+\nfToA4IYbboDH40FHR0fwfY5jz4n9eRaLPVlZWVF/RkIsWU+bNg1VVVUAgBMnTsBmsyE1NXWAnyo+\nOJ1OrFmzBm+++SbS09MBALfddltwPHft2oXc3NyBfMS48Ktf/QoffPABfv/732P27NlYuHAhxzEK\n06dPx8GDB3Hp0iU4HA64XC6OYxS+853v4MiRIwCA2tpapKSkYPTo0fjyyy8BcByjIfZ7OGnSJBw7\ndgwtLS1oa2tDTU0NcnJyov6MhOn2tHbtWnz55ZdQqVR44YUXcMMNNwz0I8WFiooK/OY3v8G1114b\nfO2VV17BihUr4PF4MHToUKxatQo6nW4AnzK+/OY3v8GwYcMwffp0PP300xzHHtq+fTsqKysBAP/2\nb/+GCRMmcBx7qK2tDcuXL0djYyM6Ojrw+OOPw2q14vnnn8elS5cwadIkPPPMMwP9mDHr+PHjWL16\nNWpra6HVanHVVVdh7dq1WLZsWbffw08//RSbN2+GSqXC/Pnz8b3vfS/qz02YgExERBTPEmLJmoiI\nKN4xIBMREcUABmQiIqIYwIBMREQUAxiQiYiIYgADMhERUQxgQCYiIooBDMhEREQx4P8HP5WxNcUI\nROYAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f1e1b0928d0>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "Qwn29SjK-XCg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Scikit-learn implementation"
      ]
    },
    {
      "metadata": {
        "id": "-kSEp8MY-y9C",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Note**: The `LinearRegression` class in Scikit-learn uses the normal equation to solve the fit. However, we are going to use Scikit-learn's `SGDRegressor` class which uses stochastic gradient descent. We want to use this optimization approach because we will be using this for the models in subsequent lessons."
      ]
    },
    {
      "metadata": {
        "id": "uKmBKodpgHEE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Import packages\n",
        "from sklearn.linear_model.stochastic_gradient import SGDRegressor\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.model_selection import train_test_split"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WuUQwD72NVAE",
        "colab_type": "code",
        "outputId": "5a3587aa-4828-4c44-eb5d-2d39f570f905",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Create data splits\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    df[\"X\"].values.reshape(-1, 1), df[\"y\"], test_size=args.test_size, \n",
        "    random_state=args.seed)\n",
        "print (\"X_train:\", X_train.shape)\n",
        "print (\"y_train:\", y_train.shape)\n",
        "print (\"X_test:\", X_test.shape)\n",
        "print (\"y_test:\", y_test.shape)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "X_train: (75, 1)\n",
            "y_train: (75,)\n",
            "X_test: (25, 1)\n",
            "y_test: (25,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "MJVs6JF7trja",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We need to standardize our data (zero mean and unit variance) in order to properly use SGD and optimize quickly."
      ]
    },
    {
      "metadata": {
        "id": "VlOYPD5GRjRC",
        "colab_type": "code",
        "outputId": "207a408e-af47-433e-e694-d237a643ff62",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Standardize the data (mean=0, std=1) using training data\n",
        "X_scaler = StandardScaler().fit(X_train)\n",
        "y_scaler = StandardScaler().fit(y_train.values.reshape(-1,1))\n",
        "\n",
        "# Apply scaler on training and test data\n",
        "standardized_X_train = X_scaler.transform(X_train)\n",
        "standardized_y_train = y_scaler.transform(y_train.values.reshape(-1,1)).ravel()\n",
        "standardized_X_test = X_scaler.transform(X_test)\n",
        "standardized_y_test = y_scaler.transform(y_test.values.reshape(-1,1)).ravel()\n",
        "\n",
        "\n",
        "# Check\n",
        "print (\"mean:\", np.mean(standardized_X_train, axis=0), \n",
        "       np.mean(standardized_y_train, axis=0)) # mean should be ~0\n",
        "print (\"std:\", np.std(standardized_X_train, axis=0), \n",
        "       np.std(standardized_y_train, axis=0))   # std should be 1"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mean: [8.22952817e-17] 7.350601608872391e-17\n",
            "std: [1.] 1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "CiE3oLCkOCEa",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Initialize the model\n",
        "lm = SGDRegressor(loss=\"squared_loss\", penalty=\"none\", max_iter=args.num_epochs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sGH_pQaDOb49",
        "colab_type": "code",
        "outputId": "ca939389-101b-4320-8115-bf0bb5aee36e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        }
      },
      "cell_type": "code",
      "source": [
        "# Train\n",
        "lm.fit(X=standardized_X_train, y=standardized_y_train)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/stochastic_gradient.py:183: FutureWarning: max_iter and tol parameters have been added in SGDRegressor in 0.19. If max_iter is set but tol is left unset, the default value for tol in 0.19 and 0.20 will be None (which is equivalent to -infinity, so it has no effect) but will change in 0.21 to 1e-3. Specify tol to silence this warning.\n",
            "  FutureWarning)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SGDRegressor(alpha=0.0001, average=False, early_stopping=False, epsilon=0.1,\n",
              "       eta0=0.01, fit_intercept=True, l1_ratio=0.15,\n",
              "       learning_rate='invscaling', loss='squared_loss', max_iter=100,\n",
              "       n_iter=None, n_iter_no_change=5, penalty='none', power_t=0.25,\n",
              "       random_state=None, shuffle=True, tol=None, validation_fraction=0.1,\n",
              "       verbose=0, warm_start=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "metadata": {
        "id": "fA8VkVVGPkTr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Predictions (unstandardize them)\n",
        "pred_train = (lm.predict(standardized_X_train) * np.sqrt(y_scaler.var_)) + y_scaler.mean_\n",
        "pred_test = (lm.predict(standardized_X_test) * np.sqrt(y_scaler.var_)) + y_scaler.mean_"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "T8Ws-khqJuNr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Evaluation"
      ]
    },
    {
      "metadata": {
        "id": "Y2pha3VRWd2D",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "There are several evaluation techniques to see how well our model performed."
      ]
    },
    {
      "metadata": {
        "id": "abGgfBbLVjJ_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RKm8IiP7O66e",
        "colab_type": "code",
        "outputId": "d0c6fd11-0c00-413f-9802-6b375addc134",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Train and test MSE\n",
        "train_mse = np.mean((y_train - pred_train) ** 2)\n",
        "test_mse = np.mean((y_test - pred_test) ** 2)\n",
        "print (\"train_MSE: {0:.2f}, test_MSE: {1:.2f}\".format(train_mse, test_mse))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train_MSE: 29.14, test_MSE: 35.00\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "TegkJM2-YKEq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Besides MSE, when we only have one feature, we can visually inspect the model."
      ]
    },
    {
      "metadata": {
        "id": "gH5N-U7YQVgn",
        "colab_type": "code",
        "outputId": "f443d734-66f4-49f2-ead9-e6b45bf1a25d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 335
        }
      },
      "cell_type": "code",
      "source": [
        "# Figure size\n",
        "plt.figure(figsize=(15,5))\n",
        "\n",
        "# Plot train data\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.title(\"Train\")\n",
        "plt.scatter(X_train, y_train, label=\"y_train\")\n",
        "plt.plot(X_train, pred_train, color=\"red\", linewidth=1, linestyle=\"-\", label=\"lm\")\n",
        "plt.legend(loc='lower right')\n",
        "\n",
        "# Plot test data\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.title(\"Test\")\n",
        "plt.scatter(X_test, y_test, label=\"y_test\")\n",
        "plt.plot(X_test, pred_test, color=\"red\", linewidth=1, linestyle=\"-\", label=\"lm\")\n",
        "plt.legend(loc='lower right')\n",
        "\n",
        "# Show plots\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2sAAAE+CAYAAAATaYj9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xtgk9X9P/B3kuZCaXpJm4JQVASq\nfOVWxAs4RGqRwvfLqHIpdjrRehuoYBGGCFrUeQeHX5koDspAZrfuNwbfDVEGTudGXW1FQEcpToUK\nNGlD05ImDWl+f5Q8fZI8SUvb3N+vf9bnyZOH47Hu8Dnncz5H5nQ6nSAiIiIiIqKwIg91A4iIiIiI\niMgbgzUiIiIiIqIwxGCNiIiIiIgoDDFYIyIiIiIiCkMM1oiIiIiIiMIQgzUiIiIiIqIwFBfqBhBF\nsqeffhrl5eUAgBMnTiA9PR1qtRoAUFZWhoSEhC69Jzc3F9u2bUNaWlrA2kpERBRsvTVOuvzud7/D\n3Llze72dROFKxnPWiHpHdnY2Xn75ZYwbNy7UTSEiIgo7PR0n7XY7brzxRnz22We93DKi8MU0SKIA\nueuuu/Daa69h2rRpqKyshNFoRGFhIXJzc5GdnY3NmzcLz1555ZU4ffo0ysvLkZ+fjzVr1mDatGnI\nzs7moERERFHphx9+wAMPPICpU6di6tSp+OSTTwAA58+fxxNPPIHc3Fzk5OTg0Ucfxblz5zB//nyY\nzWbk5ubihx9+CHHriYKDwRpRAB0+fBh//vOfMXbsWLz55pvIyMjA+++/jy1btmDNmjU4deqU13e+\n+uorjB49Grt370ZBQQHefPPNELSciIgosJYtW4ZRo0Zhz5492LBhAx5//HE0Njbio48+Ql1dHXbv\n3o0PP/wQl19+OQ4ePIjnn38eSqUS77//PgYMGBDq5hMFBYM1ogCaNGkS5PL2/8xWrlyJVatWAQAG\nDRoEvV6PkydPen2nb9++yMnJAQBcffXVnD0kIqKo09TUhIqKCsyfPx8AMHjwYIwZMwYff/wxdDod\njh49ir/+9a9oaWlBUVERJkyYENoGE4UIC4wQBVBSUpLw86FDh4TVNLlcDoPBgLa2Nq/vaLVa4We5\nXC75DBERUSRramqC0+nE7NmzhXsWiwU33XQTZsyYgRUrVqCkpARLly7FLbfcgqeffjqErSUKHQZr\nREGydOlS3H333bjjjjsgk8kwceLEUDeJiIgoJNLS0iCXy7Fjxw5oNBqvz6dPn47p06fDZDLhiSee\nwObNmzFz5swQtJQotJgGSRQk9fX1GDFiBGQyGf74xz+ipaUFFosl1M0iIiIKOpVKhYkTJ+K9994D\n0L6q9sQTT+DMmTP4/e9/j7feegsAkJKSgsGDB0MmkyEuLg4Oh4NjJ8UUBmtEQbJo0SIsXLgQM2bM\ngMViQX5+PlatWoXvv/8+1E0jIiIKumeffRb/+Mc/kJubi9tvvx2XX345+vXrh5ycHFRVVeHWW2/F\ntGnT8N133+Huu+9G//79MWrUKEyaNAlffvllqJtPFBQ8Z42IiIiIiCgMcWWNiIiIiIgoDDFYIyIi\nIiIiCkMM1oiIiIiIiMIQgzUiIiIiIqIwxGCNiIiIiIgoDIX0UGyDoanH70hJiYfJxPM2PLFfpLFf\npLFfpLFfpHW3X/R6bQBaE726M0byd9Y/9o9v7Bvf2Df+sX9862rf+BsfI35lLS5OEeomhCX2izT2\nizT2izT2izT2S/jivxv/2D++sW98Y9/4x/7xrTf6JuKDNSIiIiIiomjEYI2IiIiIiCgMMVgjIiIi\nIiIKQwzWiIiIiIiIwhCDNSIiIiIiojDEYI2IiIiIiCgMMVgjIiKfbHYH6kwW2OyOUDeFiIgo5oT0\nUGwiIgpPjrY2lO6rQVW1AQ1mG3SJamRl6pGfPTTUTSMiIooZXFkjIiIvpftqsLfiJOrNNjgB1Jtt\n2FtxEqX7akLdNCIiopjBYI2IKMZ5pjra7A5UVRskn62qNsLaej6YzSMiIgpPzc2A0xnQP4JpkERE\nMcpXquPkrIFoMNskv2NqssJktnHwICKi2OV0Imn2j6H65G+o/+JrtA0YGLA/iuMtEVGMcqU6urhS\nHR2ONugS1aiXCNhStBqkJKrR1NgSzKYSERGFhM3uQGOzDUkJaqiVCsi//w6p40YCAMxvvhPQQA1g\nsEZEFJP8pTp+ebwBo4am4aPPT2Dd1iKo7VY8WLgBAJCVmQaNKg5NwWwsERFRkEllnzzw1U5M+N2b\nAABj9Xew9k1Eo8kiBHKBwGCNiCgGNTbb/KY6ztDUo+i12wEAz9y2EqmJGmRlprEaJBERxQRx9onK\nbkNJcR4A4Muc2Ujf9s6FQO4rr4rJCnnvlgRhsEZEFIOSEtQ+Ux1XvP8ahq35GwDgZM0PmOWQB3TW\nkIiIKJyIs0+yvq3CM/9vNQDgkbteQ9OQ4Rj1YTX2V/0gPO/aRgAABTmZvdoWBmtERDFIrVQgK1Pv\ntmdNbzZg0zv3AwCann8Z1vseghpAeojaSEREFAqNzTY0NFqx67XbAADGhFQU3vc22uQKyJqsqDpm\nlPxeVbURsyYN6dXJTQZrREQxypXSWFVtxNQ9m3HHP0sBAGcO10CezhCNiIhik/5QBXZeCNQ+v3ws\nim9/Svgsua8apmbf2wgam21IT4nvtbYwWCMiihKeFas6o5DLcedVfbCoIBcA0FT4IKwvvMIDOImI\nKGZ4jp2pgwdAfq4ZAHDfvRtwJrm/2/NjMtPwZY3RZ8XkpAR1r7aPwRoRUYTzdV6aeKOzVCCXNGsG\nVJ+0701r/M17aM2dHrJ/BiIiomDyHDszlK341Utzhc9Pnz6LkftqcL7aCFOTFSnajkJbCrnMbRuB\nS1ZmWq/v72awRkQU4Xydlwa0pzp6BnLXXqbFwnt+JDxvONMIyGRBb3eka2lpwfLly1FfXw+bzYYF\nCxZgz549OHLkCJKTkwEAhYWFuPnmm7Fz505s2bIFcrkcc+fOxZw5c0LceiKi2CYeO+/+eAtmV/wR\nALDnoWKMfaYICrQXC5k1aYjXZKd4G4FnINfbGKwREUUwf+elVVUb4WhzYn9lrXAvd/cmzCv/PQDg\n3BOrYHlsaVDaGY3279+PESNG4P7770dtbS3uvfdeZGVloaioCJMnTxaes1gsWL9+PcrKyqBUKjF7\n9mxMmTJFCOiIiCi4hLHT6RSKiABA3qIyJKck4Gq7QwjM1EqF1x40hVzuM5DrbQzWiIgimL/z0hrM\nVnxRfaFilceAdP+TO/H0gono3cz62DJ9ekfa6KlTp9CvXz/J5w4ePIiRI0dCq9UCAMaOHYvKykpk\nZ2cHpZ1EROSusdmG0Qfex2PvrxPuzSjaAeDiioRIBXK9jcEaEVEE83deWlKCCmebbZj3z1L85J+/\nBQCUX3Etnst7EnJrW69XrIpV8+bNw+nTp7FhwwaUlJRg27Zt2Lx5M1JTU7Fq1SoYjUbodDrheZ1O\nB4NBejWUiIgC7+or++PqCz8/VvAqavp3pC8GokhIT3QarDEnn4gofEmdl+aSNSwNRXddJ1w/NH89\nanUDAYTfYBTJ3nvvPXz99ddYunQpVqxYgeTkZAwfPhxvv/023njjDWRlZbk973Q6u/TelJR4xMVd\nfFqNXq+96O/EEvaPb+wb39g3/kVM/5w6BQwYIFy6VtPEbhw9ABkDei9Nvad902mwxpx8IqLwJrXR\neWbLMdx2V57wjOeAFIiKVbHm8OHDSE1NxSWXXILhw4fD4XAgMzMTqampAIDs7GwUFxdj6tSpMBo7\nDlCtq6vDmDFjOn2/yWS56Dbp9VoYDE0X/b1Ywf7xjX3jG/vGv0jpn+Scm6D88gsAgOXBhTCv/gVy\n9tV4FQmZMf7SXvvn6Wrf+AvoOg3WmJNPRBS+XCX5Z00aImx0vvrKjjNhTJu24jea4UgNQsWqWFNR\nUYHa2lo8+eSTMBqNsFgseOqpp7B8+XIMGjQI5eXlGDZsGEaPHo2VK1fCbDZDoVCgsrISK1asCHXz\niYhig9MJfb8k4dJwwgCo1X6rPYaTLu9ZY04+EVH4kDpb7Zb4Jty1rOOMGEOdGQBQAIT9YBSJ5s2b\nhyeffBIFBQWwWq146qmnEB8fj8WLF6NPnz6Ij4/HCy+8AI1GgyVLlqCwsBAymQwLFy4UJjaJiChw\n1L9/D4kLHxCuXeOi2zNBKBLSEzJnV5PnAXz99ddYtmyZV07+6dOnkZWVhUOHDgmzha+99hoGDBiA\n/Px8n+87f97RrXx8IqJYZW09D5PZhh1/q8Ff/vGtcH/X2o6URyxZArz6avAbR72qO2k4kZKOFCrs\nH9/YN76xb/wL1/7RpycKP5ve34fzY8cFvw3BSIMMZE5+d/LxPYXrL0iosV+ksV+ksV+khVO/eK6k\nuc6wVtutKPvfecJz9676M555aALUAWx3d/slYjagExFRxJKdOYO0kcOEa6nVtEgi7+yBiooKbNq0\nCQDccvJPnDgBAG45+YcOHYLZbMa5c+dQWVmJceOCH8ESEUWj0n012FtxEvVmG5wA2pztq2niQG1G\n0Q7UW+xobJY+d42IiCiaJU+9WQjULA8uiPhADejCyhpz8omIQstmd6Cq2n0PsDjt8d77NsKQqAfA\nkvxERBRdXIW0XGOb5P5rH0VEokGnwZpGo8GaNWu87v/hD3/wupebm4vc3NzeaRkREQFoH5gaLhx6\nvfbdJRh25rjwGUvyExFRNBKn/9ebbdCo5ABksLU6oEtUIytTj/zsoYj/w+86LSISybpcDZKIiEIj\nKUENXaIaJcXThHuvTC/Cx1fdBLkMcDoBXSJL8hMRUfRwpf+7WFvbhJ/rzTbsrTiJRQUdW65CVUQk\n0BisERGFuaStm1BSvES4Fq+mTcoaiKnXDmJJfiIiihpS6f9iyedM2PrWPcJ1tK2miTFYIyIKM+L8\n/IyBKcL9M4OH4+d3vwa5x+HWCnmntaKIiIgihjj939Oad5ci88wxAMD/Zf03Br/3a6QHs3FBxmCN\niChMiPPzE4//G69vfUz47MjR00hKUOM5+NhcTUREFCVc6f/14oDN6cSu124TLm9/9HdI1CXiuSgv\nqsVgjYgoTLjy890OuAYwZ/lO2N464LahmqtpREQUrdRKBbIy9cKetdmf/QF3/32r8LlrO0AsFNVi\nsEZEFCTi9EbPwcVmd+DQV7VugdodP9uK5j5a4MKmateGagAoyMkMXsOJiIiCzFUwS1xE5BdznsZn\nl2YhVRs7RbUYrBERBZg4vbHBbJNcIcsYmIJ3RN/xLMkvVlVtxKxJQ6J+NpGIiGKXsta92uPJWhPu\nBpAXY1sBGKwREQWYZ/lhzxUyfXqi8Flx3kp8foX/0sOmJisam21IT4kPTIOJiIhCSDwu2q8fj7O7\n9sC1My3Wxj4Ga0REAeSv/HDOk/dBX/CFcL1uewU+FwV1vqRoNUiK8g3VREQUg5xO6PslCZeGb08D\n8bEVnHniDnUiogDyVX5419o8XP2f9kDNOmsuDHVm5GcPRc64DKQmaiCXARqVdIpHLGyoJiKiyGaz\nO1BnssBmd3Tp+T7/+0v3QK3OHPOBGsCVNSKiXuVZRMSz/PCkr/+Gx3e/Jjx/stYkBF4KuRwFOZmY\nNWkIGpttSIhXYscn/0FVtREmj7PViIiIwlFX9ml7Eqc9Nm7/PVpzpgaruWGPwRoRUS/wNzi5yg97\nluRft70CBRIrZGqlQsjJFwdvsbShmoiIIlNn+7TFFN/UQHfDWOHaUGcOTiMjCIM1IqJe4G9wumOY\nCosKOgK1e576M8Zc1a/LK2Ti4I2IiChc+dun7VnJWLya1qbW4IdvToG7sb0xWCMi6iF/g5O47DAA\nHDl6Gs9yhYyIiKKQr33agHslY3Gg9tDyMvzQGgfdxgOdpkvGIvYEEVEP2OwOfFPbKOxJEzidbmmP\nDZ9WwFBnRnpKPAM1IiKKSq592lJStBpc+tJTboHajKIdqG2NgxMdGSml+2qC1NrIwJU1IqJu8Nyj\nJpcBbc72zzz3pomLiBAREUUrtVIh7NP2VFKcK/zc8Ms3UXT2ckBiFc4zXTLWcWWNiKgbXHvU6s02\nOCEdqG350V1Yt72CAw4REUU9V6n+vIlXuB1D81/2erex0VBnxqlpt3WaLkntuLJGRHSRpPaoPVP2\nNLK+Pyhczy9+n2X2iYgo6jkcbdi+t9qrGvLqwmtx2aBU4TmnSgXjSSMAeB1rI5ai1SApgaVGXBis\nERFdJM8N1J5pjx9/9i2eG5jEFTUiIop6m3YdkayGLC6wZTx+Ek5tx141f+mSWZlpHD9FGKwREV0k\n14zg8H/9FT//86vC/RlFO5CaqGGgRkREMcFmd+DA4VNu957Y+SIm1BwQrn2dnebKPKmqNsLUZEWK\nVsOMFAkM1oiILpJaqUBJ8TS3ezOKdgDgjCAREcWOxmYbDGdbhGtxpsmmSfNx/dsvIt3HdxVyOQpy\nMjFr0hA0NtuQxGNtJDFYIyK6CDJTA9KuvFy4XvDzMtSej0MqZwSJiCjGJCWooU/uA+2/D+OX7y4R\n7rsyTW7twt4ztVKB9JT4QDYzojFYIyLqIvHZMEB7ascKu4MzgjGqpaUFy5cvR319PWw2GxYsWICr\nrroKy5Ytg8PhgF6vxyuvvAKVSoWdO3diy5YtkMvlmDt3LubMmRPq5hMR9ZhaqcCvV011u8dMk97F\nYI2IqAvEgVrjr7eidcZMAJwRjGX79+/HiBEjcP/996O2thb33nsvxo4di4KCAkybNg1r165FWVkZ\n8vLysH79epSVlUGpVGL27NmYMmUKkpOTQ/2PQETUI+Kx8eGlpTjRpmamSS/rNFjjzCERxSKb3YFT\nxnO4RGI1jQgApk+fLvx86tQp9OvXD+Xl5Vi9ejUAYPLkydi0aRMGDx6MkSNHQqvVAgDGjh2LyspK\nZGdnh6TdREQ9lTwtG8rPK4RrQ50Zy5lpEhCdHortmjnctm0bfvnLX+LFF1/E66+/joKCAmzfvh2X\nXXYZysrKYLFYsH79epSUlGDr1q3YsmULzp49G4x/BiKiXuNoaz8vZuXGA7hEnyDcbx0zloEaSZo3\nbx4ef/xxrFixAi0tLVCpVACA1NRUGAwGGI1G6HQ64XmdTgeDweDrdUREYU2fnigEapZHHgOcTgAd\nmSYM1HpXpytrnDkkolhSuq8G1zz9KBYd+4dwb0bRDuSMy0BBCNtF4eu9997D119/jaVLl8J54S8t\nANx+FvN131NKSjzi4i7+Lz16vfaivxNL2D++sW98Y98AqKoCxo7tuHY64doEwP7xrad90+U9a/Pm\nzcPp06exYcMG3HPPPZw5JKKoY7M73A7xBDo2SldVGzFr0hDOGJLg8OHDSE1NxSWXXILhw4fD4XCg\nb9++sFqt0Gg0OHPmDNLT05Geng6j0Sh8r66uDmPGjOn0/SaT5aLbpNdrYTA0XfT3YgX7xzf2jW/R\n2De2i0xZlCqwhQt9Eo3901u62jf+ArouB2uBmDns7qyhJ0bz0tgv0tgv0mK9X1o//CtUt+YI1zMe\n+yMgkwnXpiYrFCol9Gl9Q9G8sBPrvy8AUFFRgdraWjz55JMwGo2wWCyYOHEi9uzZg5kzZ+KDDz7A\nxIkTMXr0aKxcuRJmsxkKhQKVlZVYsWJFqJtPRDHI0daG0n01qKo2oMFsgy5RjaxMPfKzh0Ihl94d\nJQ7UjF99A2daWrCaS+hCsBbImcPuzBp6YjQvjf0ijf0iLZb7xdHWhv79k6ES3XOtpomlaDVwtNpj\ntp/Euvv7Em0B3rx58/Dkk0+ioKAAVqsVTz31FEaMGIGf//znKC0txYABA5CXlwelUoklS5agsLAQ\nMpkMCxcuFLYMEBEFU+m+GuytOClc15ttwnVBTqbbs8n/PQXKf5UL19y3HRqdBmucOSSiaOGV9tHa\niv4ZHTOEC3/6Or5Pu1TyuzwvhjxpNBqsWbPG6/7mzZu97uXm5iI3NzcYzSIikmSzO1BVLb1FyTPV\nX7yaZlm4COeefjYobSRvnQZrnDkkokgnlfZRUjzN7RnP1TT5hQzIFJ4XQ0REUaCx2YYGs03yM1OT\nFY3NNlxysga6W34k3OdqWuh1Gqxx5pCIIp1n2oc4UNt79S1YN/URr+84ncCzD01Aal8lV9SIiCji\nJSWooUtUo14iYEvRanD1lf3d7jFQCw9dLjBCRBSJxGkfu9bmuX02v3h3ezGkplav7+kSNbjyshQ0\nNbYEpZ1ERESBpFYqkJWpd5u8dCkp7lhsMR45DqdeH8ymkR8M1ogoqhlMFtSbbV6B2oyiHZCZbcga\nloaGJqPX97Iy06BRxYHlRIiIKFq4Uvqrqo0wNVnx8u+fxJXfHxE+52pa+GGwRkRRybVPbdSap7Dr\n878I98V702QyoOqYERpVe5qjrdUBXSL3qBERUXRSyOUoyMnErElDkDEwRbhv+dkjOLf6FyFsGfnC\nYI2IolLpvhqfB1y7tF04DtLa6gAA3DiiP+6ceiX3qBERUdRS/PtrZNx0vXDN1bTwxmCNiKKO4/Bh\nLCqYIFzPefi3sKr6CNcyAE6J7/37+7OBbxwREVGIiEvyAwzUIgGDNSKKKp4Dkedqmq9ADegoXZye\nEh+YxhEREYWIeHw0fvUNnGlpfp6mcMFgjYiig9MJfb8k4fKdmUX405CbvB7TJarhdDrRIFEBMkWr\nQVKCOqDNJCIiCqbEu/Kh3rNbuOZqWmSRh7oBREQ9pU9PdAvUDHVmnMsvkHw2K1OPsVem+/gsjfvV\niIgoaujTE4VA7dxjjzNQi0BcWSOiiOYr/96zPHGK1rvKo7/PiIiIIpXi6L+hm3idcM0gLXIxWCOi\niOQZpP24aAd0iWpk7a1GfvZQt/LEjc02JCWo3VbN/H1GREQU7mx2h+QYxiIi0YXBGhFFBPGgJD4b\nBugoIlJvtmFvxUkA7cEYAKiVCp8FQ/x9RkREFI5c54hWVRvQYLa1T1Rm6pGfPRT9+ycLzxmPHIdT\nrw9hS6k3MFgjorAhNUsoHpSu/ftOLNz7pvD8/OLdqDfbvN5TVW3ErElDuFpGRERRp3RfjTAxCbRP\nVF6/agH6F3wm3ONqWvRgsEZEIedvltA1KO1am+f2naff/Dvqjxkl38cS/EREFI1sdgeqqg1u98Tj\no/nhx2B7anWwm0UBxGCNiEJOapZwb8VJONqcqDn0H+xaO1f47L57N+BMcn/gmBFyGdAmcWgaS/AT\nEVE0amy2oeFCRklGw0m8WfKw8NnMJTvw/AM3QLreMUUqBmtEFFJSs4QuRXde63btecC1VKAGsAQ/\nERFFp6QENXSJapQUT3O7P6NoB1I5URmVGKwRUUD5qlblIp4lFBOndewZMQVv3LrQ558hlwFOJ6BL\nZAl+IiKKXmqlwi1Qu+vBEpzt215UhBOV0YnBGhEFhL99aAq5XHjONUvoKhTiuTdt7dbPsL/qB79/\nlhPA4/PG4IqBSRyoiIgoKiXO/wnUf9klXM8vfh/mJitSeVZoVGOwRkQB4WsfGtBRVh9onyXMytRL\nFhFZt70CBdlDoVDIUXnUgIYm7xU4ANBpNQzUiIgoaonPTrM8vBjnnnoGz3WSuULRgcEaEfU6f/vQ\npMrqL3jxPiz68gvhen7x+8Isofhw6217juLTw6e93snUDyIiikaKmmPQTbhGuBaX5OdZobGBwRoR\n9Tpf+9AA77L64tlCADhy9DSek5glVCsVmD/9KvTRxKGq2ghTkxUpTP0gIqIo5Tk+8uy02MRgjYh6\nnec+NDFXWX3lp58g+bb/Fu4bausBpdJvyWHxKhtTP4iIKFqJAzXj4Ro401mQP1bJO3+EiOjiuPah\nScnKTEPGwBT3QK3ODCiVF/X+9JR4BmpERBRV+j65zC1QM9SZGajFOK6sEVFAuFITxSmLY4em4NGC\nccIzxt+U4sz1k5BkdzDwIiKimOZWROSRx3Bu1eoQtobCBYM1IgoIz5TFq6/s7/b5uu0V7WX9vzjg\ns6w/ERFRtJN/9y1Srx0lXJ+sNbWn+nMik8BgjYgCTK1UuAVqjksvwxsv/qFLZf2Jwt3LL7+Mzz//\nHOfPn8eDDz6Iffv24ciRI0hObj+ktrCwEDfffDN27tyJLVu2QC6XY+7cuZgzZ06IW05E4cCziMi6\n7RWo2njA7/mkFFu6FKxxMCKii2G7cPaL52qaoc7cXtZ/4wHJ70mV9ScKVwcOHMCxY8dQWloKk8mE\n2267DTfccAOKioowefJk4TmLxYL169ejrKwMSqUSs2fPxpQpU4QxlIhik1sRka++wbtfNHAik7x0\nGqxxMCKirnK0taF0Xw2qqg0oKZ7m9pmr5PDFlPUnCmfXXnstRo1qT11KTExES0sLHA6H13MHDx7E\nyJEjodVqAQBjx45FZWUlsrOzg9peIgoPfVctR/xbvxKuhYnM6qOSz3MiM7Z1GqxxMCKirirdV4NL\n1zyLRZ//Sbg3o2gHcsZloODCdVfK+hNFAoVCgfj49omFsrIy3HTTTVAoFNi2bRs2b96M1NRUrFq1\nCkajETqdTvieTqeDwSB9aDwRRTfxalrzqmfQ8shiAJzIJN86DdYCORilpMQjLq7nswR6vbbH74hG\n7Bdp7BdpPe0Xa+t5LBJVegTaAzUA+PJ4PR6c1QcaVfv/5dw4eiB2fvKN1ztuHD0AGQPCazWevy/S\n2C8d9u7di7KyMmzatAmHDx9GcnIyhg8fjrfffhtvvPEGsrKy3J53Op1dem93x0j+u/GP/eMb+8a3\nHvfNt98Cgwd3XDudSACQcOFSm9QH+pQ+qDO1eH01LbkPhlyeKoyh4Yi/O771tG+6/G89EIORyWS5\n+BZ70Ou1MBiaevyeaMN+kcZ+kdbTfvGsZHXnQyVojO8IuoxnW3D823phVnDG+EthaWl1K+uflZmG\nGeMvDat/P/x9kdbdfonGwfyTTz7Bhg0b8M4770Cr1WL8+PHCZ9nZ2SguLsbUqVNhNBqF+3V1dRgz\nZkyn7+7OGMnfWf/YP76xb3zrad94FhEx1JkBifeNGpLqtmdNfL+psQXh+m+Hvzu+dbVv/I2PXSot\n4xqMNm7cKAxGw4cPB9A+GFVXVyM9Pd1rMErnIX5EUctmd6DOZIE+PdEtUJtRtMMtUAO80xtdZf2f\nu/96PP/ADXju/utRkJPJale6rkjzAAAgAElEQVQUUZqamvDyyy/jrbfeEvZnP/LIIzhx4gQAoLy8\nHMOGDcPo0aNx6NAhmM1mnDt3DpWVlRg3bpy/VxNRlHArInLkuLB/W0p+9lDkjMtAaqIGchmQmqhB\nzrgM4dxSik2drqy5BqOSkhK3wWjZsmUYNGiQ22C0cuVKmM1mKBQKVFZWYsWKFQH/ByCi4PJVRKT5\nsaX49bX5gMSsYFZmmuTGaLVSwRx8ilh/+ctfYDKZsHjxYuHe7bffjsWLF6NPnz6Ij4/HCy+8AI1G\ngyVLlqCwsBAymQwLFy4U9ncTUXTq+9QKxG94Q7j2F6S5eJ5PmpSgZlER6jxY42BERGKl+2ok96ZN\nGNEfBRPb8/E90xs5K0jRKD8/H/n5+V73b7vtNq97ubm5yM3NDUaziCjE3IqIPPk0WhYtuajvcyKT\nxGTOru50DoDeyG9lnqw09os09ou0rvaLze5AxsAUt3uuIiIAkHrhAM+8iVeg2dIa8bOC/H2Rxj1r\nwdHdPubvrG/sH9/YN751tW/kJ75H6jUjhOuurKZFA/7u+NYbe9bCt6wMEYWc63DrpAS13yDNhQd4\nEhFRNBOPi+LJSMkiIkS9gMEaEXkR70trMNuwc22e2+dSgZoYD/AkIqJo4jku6i5kkuRnD0X//h1F\ntf796SEkXJ4BnhhKvYXBGhF5Kd1Xg70VJ3HroQ/xyIfrhfvr3v0XIJNJFhER4wGeREQUTVzjoku9\n2YZBa55D/4KOycv5xbvR8Ifj0CWeFAI5VjmmnmKwRhQDfKVt+Hq2qtqAXRKraanH6rG68DoAQOVR\nAxqabJLv8CzVT0REFCk8x0xr63lUVRvcnhGPkX+fswAvDboVMLePidwSQL2JwRpRFPOXtuFrts9s\nMLmV5F9yx8uovqR9sDE1WdFsaRVKC2/bcxSfHj7t9Q5fpfqJiIjCla8xc9YtmWi4EIjpzQZseud+\n4Ts/LtqB5AQ10Ow9ecktAdQbGKwRRSHXrOCef53A/spa4X5ns3369EToRdeee9PEK2ZqpQLzp1+F\nPpo4luonIqKIJ5XquLfiJFSqOOgS1W4TmUD7GJmcoMJZiUAN4JYA6h0M1oiiiHhWsN5sg1wm/ZzU\nbJ+4klXtVVl4aPrTXt/zXDHjAZ5ERBQNXFsApFR8fcYtUPvJz34Dc5/2MTNrWBq+PF6PerN3wMYt\nAdQbGKwRRRHPWcE2H6coimf7pMoNK9rakLOvpssrZjzAk4iIIlljs01IdRSb+flO3Pe3TcL1/OL3\n0dxkRapoXFQo3MdeF24JoN7AYI0oSvibFfSUotUgIV7pFait216B/LY2rpgREVFMSUpQQ5eodlsh\nExcRMb30Gs7fU4jnJAp2uSYyuSWAAoHBGlGU8DUrKOWFbUtxSfEh4VrYm+axn40rZkREFAvUSgWy\nMvXYW3ESyedM2PrWPcJnG//4JfJuvFx4znNc5AQnBRKDNaIoITUr6CKXAU4AOq0GJcW5bp95FhFh\n9SoiIoo2XTnCJj97KBYVjBOuz8sVWL+tHA/PuBoNDec6/TM4wUmBwGCNKIJ5Dj6uWUFPk8YMwExN\nPYbM6QjU8h/+LSyqPl7PsnoVERFFi4s5wqZ//2Th538f+AoJgy5BgVIBhYIHW1PoMFgjikC+Bp/Z\nN18BwDtvXjxTCAAna03os/EALKxeRUREUcxXOX6gI+W/z9u/QsLK5cIzhjozUoPbTCKfGKwRRaDO\nBh8hb76vChkZOuG5phfXwHrv/VADPlfhWL2KiIiigb/CW66U/4yBKcK9pld+Cevd9wareURdwmCN\nKMJ0ZfBRKxW4+sr+bp8Z6sxu16xeRURE0cxf4a2206fdAjXPMZIoXDBYI4ow/gYf136zzgI1wL16\nlUKlhKPVzhU1IiKKGr4Kb4lL8jvlchhPnw1204i6jDsmiSKMa/CR8qc1eW6BmqHO3OlsoVqpwCVp\nfRmoERFRVHEV3hITB2rGr//DQI3CHlfWiCKIq/rjqKFp2F9Z6/aZeAACmNJBRETkSu1P3fpr3Ln7\nTeH+6dNnvapBEoUjBmtEEcCz+mOKVoVB6QmwWO2YtK8U93xcIjzLII2IiKidQi53q4hsev5VnL/v\nATCXhCIFgzWiCOBZ/bGhqRUNTa1cTSMiIvJBZmpA2pWXC9ccIykScf2XKMxJVX/UtpjdArX6zw5y\nECIiIrog+dZJQqBmH3cdx0iKWFxZIwpzntUfPVfTjhw9jfSU+GA3i4iIKCzp0xOFn43HvoczKTmE\nrSHqGa6sEYU5cfVHcaD2f6OnY37x+0hKkK4MSUREFEtUu3a4BWqGOjMDNYp4XFkjCjOuio9JCWqo\nlQqolQqUFE9ze2ZG0Q4AQE5mGkvuExFRzBMHaY2btqH1f34cwtYQ9Z4uBWsvv/wyPv/8c5w/fx4P\nPvggRo4ciWXLlsHhcECv1+OVV16BSqXCzp07sWXLFsjlcsydOxdz5swJdPuJooZnxUddohpZmXq3\nKlYAMHPJDqRqNcjKTBNKEhMREcUi2VkT0jIvE665N42iTafB2oEDB3Ds2DGUlpbCZDLhtttuw/jx\n41FQUIBp06Zh7dq1KCsrQ15eHtavX4+ysjIolUrMnj0bU6ZMQXIyl5+JusKz4uMzr9+PSxs6rg11\nZtjsDjwvWnUjotDiZCZR6CRPvRnKqkoAgP2acTi7e1+IW0TU+zoN1q699lqMGjUKAJCYmIiWlhaU\nl5dj9erVAIDJkydj06ZNGDx4MEaOHAmtVgsAGDt2LCorK5GdnR3A5hNFB8+Kj55FRE7WmqAGoFYq\nWEyEKExwMpMo8Dy3Bri4FRGp/g7O5JRQNI8o4DoN1hQKBeLj2/9yWFZWhptuugl///vfoVKpAACp\nqakwGAwwGo3Q6XTC93Q6HQwGg+Q7icidq+Lj2P9UYvUfnxHuz1z8B0ChwPPNNgZpRGGGk5lEgeNr\na8BPLUeQct/dwnNMe6Ro1+UCI3v37kVZWRk2bdqEW2+9VbjvdDoln/d1XywlJR5xcT1P5dLrtT1+\nRzRiv0gLx37RJvXBTo/VNFcREX2SBn21GmiT+kCjClxNoHDsl3DAfpHGfuFkJlEgeW4NqDfb3PZw\nN/56K1pnzAxF04iCqkt/8/vkk0+wYcMGvPPOO9BqtYiPj4fVaoVGo8GZM2eQnp6O9PR0GI1G4Tt1\ndXUYM2aM3/eaTJaetR7tf2EwGJp6/J5ow36RFpb9cv489AM6/iJXfNsqfD74GuG6yWLHo69+JMwq\n5mcPhULeu6duhGW/hAH2i7Tu9ku0BniBmMwEuj+hGa393FvYP76FS99YW8/jy+P1wnVfazPe+9Wd\nHZ/b7EgK4OSllHDpm3DF/vGtp33T6W96U1MTXn75ZZSUlAj59RMmTMCePXswc+ZMfPDBB5g4cSJG\njx6NlStXwmw2Q6FQoLKyEitWrOhR44iinTjnHgDWba/At9VGyJusUCkVsLY6YG11AGifVXTNMhbk\nZAa9rUTkLVCTmUD3JjQ5weAf+8e3cOqbOpMFBlMLAGDHa7dD4WwDABztPwzLfvIKnv+2PqhbA8Kp\nb8IR+8e3rvaNv4Cu02DtL3/5C0wmExYvXizce/HFF7Fy5UqUlpZiwIAByMvLg1KpxJIlS1BYWAiZ\nTIaFCxcK+flE5L1JWhyoOfr1R8OhahQAmDVpCAwmC9aVfSkEamJV1UbMmjSE1SCJQoyTmUSBkZSg\nhi5R7XbG6LwF23BOk4BUrQZJCeoQto4ouDoN1vLz85Gfn+91f/PmzV73cnNzkZub2zstI4oSnpuk\nPfemeW6OVisVUCkVaDDbJN9narKikQVHiEKOk5lEgZG0bTNKiouEa9cebgDIykzjZCXFlOAm/BLF\nIPEmac+S/L6qWLlmFeslArYUzioShQVOZhL1PnHWyf67l2LL4Fsgb7IiRatBVmYa8rOHhrB1RMHH\nYI0ogFznpz3ywRu49fBe4f6Moh1ISVCj2NIKbbzK63tqpQJZmXq3SlgunFUkIqJoI6uvR9rwwcK1\noc6MEQCe83HOGlGsYLBG1Is896U1Ntvccu6BjnQOU7MNT2/6DOOuSpes8OiaPayqNsLEWUUiIopS\naf2SIBNVSRVnnaiVCqb9U0xjsEbUC6QO75yotaJwyW3CMz95aAvM8Ulu3zvb3OqzwqNCLkdBTiZm\nTRrCWUUiIopK4rRH4+EaONPTQ9gaovDDYI2oF3ge3ulrNc0XfxUeOatIRETRRrNlE7RLO4rz+NrD\nTRTrGKwR9ZBrX5qLuIjI1ht/gt9dPwcAoFbKYbO3Sb6DFR6JiChWiFfTmn/xElru/1kIW0MU3his\nEfVQY7MNDWabV6VHz9W0eHUc+qjbUx89scIjERFFO6kiIkTkn7zzR4hIzGZ3oM5kgc3efmB1UoLa\n6+w0qbTHxnOtuPpyneQ7WeGRiIiiWdoAHQM1om7gyhpRF0kVEbmYvWkpWg3umJKJPpo4VngkIqKY\nwSIiRN3HYI2oE65y/Hv+dQL7K2uF+56B2rrtFUitNqLebJV8T1ZmGuLVcazwSEREMUHzm83QPr5I\nuOZqGtHFY7BGJMFmd6DBbMXeihP48ng96s02yGXtn2Uf2YfH9rwuPDv/6d147oEbUKBUYNakIaLv\nNfhcPWOFRyIiimZuRUSefQEtDy4MYWuIIheDNSIRcapjvdnm9lmbE5JFROTNNqGSo1qpwCWpfXHX\n1Ku8DsgmIiKKdrKGeqRdxb1pRL2FwRqRiOd5aS6a1hb8/o07hOtFd67FN+lXAAAS+6rQR+39nxJX\nz4iIKJakZaRB1tpR8ZiBGlHPMVgjusDzvDSXzkryn21uxTMl/0JWph752UOhkLPIKhERxRa3IiKH\njsHZr18IW0MUPfi3SqILXOeliYkDtcMD/6s97VHm/d16sw17K06idF9NoJtJREQUNjTbtrgFaoY6\nMwM1ol7ElTWiC5IS1NAlqlHfyQHXPxrVH4eOm2Bqtnm+AlXVRsyaNIR71IiIKOqxiAhR4DFYI7pA\nrVQgK1OPRQXj3O67ArXUxPaqjpOzBuKTg6cl32FqsgrFRoiIiKIRi4gQBQ+DNaILUm66Hov+/bVw\nPXPJDqRoNZg8NBU512RAl6iBWqmAze4QVuC83qHVIClBHcxmExERBU3apemQWTvOE2WgRhRYDNaI\n4J7KAQAna0143kfZfdcKnFTVyKzMNKZAEhFRVBKPlfWHqtHWr38IW0MUG1hghGJaXPkB943R/zkF\nQ51ZKLvvK/DKzx6KnHEZSE3UQC5rT5HMGZfhdvA1ERFROLLZHagzWWCzO7r0vObd33gVEWGgRhQc\nXFmjmOW5mnYxqRwKuRwFOZmYNWkID74mIqKI4GhrQ+m+GlRVG9BgtkGXqBaOnfHFrYjI6ufR8rOH\ng9FUIrqAwRrFHqcT+n5JwmXTK7+E9e57u/UqHnxNRESRonRfjVsKv+vYGQBYdMc1bs/KDAakXT1E\nuObeNKLQYLBGEc9md3R9dUsmg150eeTo6fbvBbSFREREoWWzO1BVbZD8rKraCGvreeG6J5knRNS7\nGKxRxPKXzqGQe2/H9Bx85hfvRsNbBzr9HhERUaRrbLahQaKKMdB+7IzJbEMcPIqIfH4YLf0HotFk\nYbo/UYh0KVirrq7GggULMH/+fNx5551Yvnw5jhw5guTkZABAYWEhbr75ZuzcuRNbtmyBXC7H3Llz\nMWfOnIA2nmKbv3SOgpxM4b5nkLZue0X7cxcGLV/fIyIiihZJCWq/x86kvf06lE+uEO6dPn22fUJ0\n14EuTYgSUWB0GqxZLBY8++yzGD9+vNv9oqIiTJ482e259evXo6ysDEqlErNnz8aUKVOEgI6oN3WW\nzjFr0hColQqvQM1qs6Pqhb2dfo+IiCia+Dt2pqQ4V/jZ8uBCnHv2BZTure7ShCgRBVanUyMqlQob\nN25Eenq63+cOHjyIkSNHQqvVQqPRYOzYsaisrOy1hhKJ+UvnaDBboX78Ma8yw4Y6M0xm/2kgjc3S\nnxEREUU6z2NnBstbsGttnvC5oc6Mc8++0OmEaFdL/hNRz3W6shYXF4e4OO/Htm3bhs2bNyM1NRWr\nVq2C0WiETqcTPtfpdDAYpP9DJ+opf+kcO0UDD+C+MTol0X8aSFICS40QUddxmwBFEvGxMxkDU9w/\ndDoBQxOAzve3NTbbWAmZKEi6VWBk5syZSE5OxvDhw/H222/jjTfeQFZWltszTqez0/ekpMQjLq7n\nKWd6vbbH74hG0d4vN44eiJ2ffCNcpzQ34Ddvd5Tgf+CeX+HaH/8I93v0g+f3Ou4PQMaA2E3bjfbf\nl+5iv0hjv3CbAEUucaBWX/4FWjIug8J4Dg67A2qlotP9bZzYJAqebgVr4oEpOzsbxcXFmDp1KoxG\no3C/rq4OY8aM8fsek8nSnT/ejV6vheHCTBB1iIV+mTH+UlhaWlF51IAtq6e5f1a0AwDw6cEfMO26\nQcI+NL1eK3yvqtoIU5MVKVoNsjLTMGP8pVHfZ77Ewu9Ld7BfpHW3X6ItwHNtE9i4caPf58TbBAAI\n2wSys7OD0UwiQfxLv0DfNS8J10IRkd0H0NBkg07bUUTE1/62rMw07u0mCqJuBWuPPPIIli1bhkGD\nBqG8vBzDhg3D6NGjsXLlSpjNZigUClRWVmLFihWdv4yom1zpHIsKxgn3PrrqJqyZXiRcS6VriNNA\nunw+GxGRh0BvE+hu9km0BcW9LWb7Rybr+Pmee4BNm7BrxyHJIiLxfVR4eG4W4vuocODwKRjPtiAt\nuQ9uGHEJ7p1xNRSK2KsGGbO/N13E/vGtp33TabB2+PBhvPTSS6itrUVcXBz27NmDO++8E4sXL0af\nPn0QHx+PF154ARqNBkuWLEFhYSFkMhkWLlwozCISBYJnpUfXapqYv3QNtVLBnHsi6lW9tU0A6F72\nCVeD/YvF/pEZDEi7eohw7drHbfvhLD49WCv5HVdWSt6Nl2PadYPcJjYbGs4Fpd3hJBZ/by4G+8e3\nrvaNv4Cu02BtxIgR2Lp1q9f9qVOnet3Lzc1Fbm6u132ii2WzO/yuekmdnQamaxBRiPXWNgGirrjY\nsVJccKurRUQ4sUkUWt1KgyQKFEdbW3v+fLVB8hBOXwNPflsbAHjtQ8vPHhr0fwYiil3cJkDB0NlY\nCbgHavXlX6Bt8BVu72AREaLIwGCNwkrpvhqfh3CK96YBwMlaE1xDCfehEVGwcZsAhYq/sfK+yjL0\nffVF4TPxapqYv0OymZVCFD4YrFHIeKZv+DqEc0L1P7Bo7cvC9T1P/R/qm89Dt/GA10wi0zWIKFi4\nTYBCwd+B1eJJTeu8n6Dp9Tf9vsuVfcKsFKLwxWCNgs5X+sbkrIFe+fO7PA64nlG0A2g+D8B9JrEg\nJzM4jSciIgohqb1miZZGvLvhbuHa12qaJ3FWikKlhKPVzhU1ojDDYI2Czlf6hsPRJuTPKxznsWPd\nbOGZ1wpW44th1wJNrV7vq6o2YtakIRxgiIgo6nnuNfOc1BRvEegqtVIBfVpfVvQjCkMM1iio/KVv\nfHm8AaOGpqHozmvd7s8o2oEbR/SH6fBpye9JnaVGREQUjcR7zcSB2gP3/ApXT7kOBZy4JIoqsXeq\nIQWVze5AnckCm90BoPNSweJAzZiQivnF7yNnXAbumJIJXaL0XCGrVhERUSyZf+Jjt0BtfvH7uHrK\nddxrRhSFuLJGAeFrX1rexCskSwVLpXE0NtvwnKiqI6tWERFRrBOX5Dc/vBgnHlnuNlYSUXRhsEYB\n4a+ssGfQ5RmoGerMUANeaY2sWkVERLFKdtaEtMzLhGtXEZH0UDWIiIKCwRr1On/70qqqjVhd2J7q\neN0LSzH+0H7hs9Onzwol+KXwLDUiIopF4tU0oOvVHoko8jFYo17nb19aQ5MV351q8jrg2lBnRlfD\nLp6lRkREsUIcqNV/fhhtgy4NYWuIKNhYYIR6naussJSBDbW46brLheszX/+HM4REREQeVH/e5Rao\nGerMDNSIYhBX1qjXicsKi0kdcJ1TVY+CnNRgNo+IiCisiYM007Mv4Ye5dyPJ7mDqP1EMYrBGASEu\nBtLQZMXONR2B2qab5uOP4/KEz3mgNRERESBrbkLaFQOF63XbK9qrKr91QKiqnJ891O/+biKKLgzW\nKCBcxUA896bNKNrhds0DrYmIiIDk7B9BefhLAIBTocDrW8t9VlUuyMkMSRuJKPg4NUMB41m9yjNQ\nA3igNRERkT49UQjU6r88itrvjX6rKtvsjmA2j4hCiMEa9Tp9eqLXpuh12yskn+WB1kREFAtsdgfq\nTBa3QEuyiEj/S/xWVXZlpBBRbGAaJPUqX2fB8EBrIiKKRY62NpTuq2nfe2a2CXvPxNsEmn/xElru\n/5lw7aqqXC8RsDEjhSi2MFijXqF+710kPtox0BjONAIymXDNA62JiCja2ewOrzGudF+N294zi/Es\nFhVPE66ljq/xVVUZYEYKUaxhsEY95ms1TQoPtCYiomjja/Usb+Jgt71na99dgmFnjgMA2mRy/HCy\nHr7WyJiRQkQAgzUSkZoR9KetyYx+QzKE6xULNqDfj8Yhv62NZYWJiChmeK6euSo3Wqznhb1n4rNG\n777/1zibmIrn/VRDZkYKEQEM1gi+ZwT9neXiq9LjIZYVJiKiGGKzO3xWbvz3dyZcbzqGJzcvFe65\nxsvULu49Y0YKUWxjsEY+ZwQdbU5MvXaQMJi4ZvYyBqYIz+6/ahLWTn/M7X086JqIiGKFv8qNW1Z3\n7E174X+W4R+ZE4Rr7j0joq5gsBbj/M0I/q2qFvsra6FRyQHI8PsXZ7h9/uOiHXBKfI8HXRMRUayQ\nqtyostvwh//NF67Xba/A0Woj5Nx7RkQXicFajPM3I9h2IRKztra55doDwNpt/4KuxsiywkREFNM8\nKzcu/PBXyD30AQDg5PCxUP/tIxQA3HtGRN3SpWCturoaCxYswPz583HnnXfi1KlTWLZsGRwOB/R6\nPV555RWoVCrs3LkTW7ZsgVwux9y5czFnzpxAt596yN9ZLgBQ/P+ewTXfVgrXQq59TT1GDUnF/qof\nvL7D1A4iIoolrlUy8dlpb2/4ED/Ou1a45t4zIuqOTkv2WSwWPPvssxg/frxw7/XXX0dBQQG2b9+O\nyy67DGVlZbBYLFi/fj1KSkqwdetWbNmyBWfPng1o46nnXDOCUnatzZMM1ID2VMeccYOQMy4DqYka\nyGVAaqIGOeMymNpBREQxRfXNcbdA7WStCbfdfj0rIxNRj3W6sqZSqbBx40Zs3LhRuFdeXo7Vq1cD\nACZPnoxNmzZh8ODBGDlyJLRaLQBg7NixqKysRHZ2doCaTr1FfJZLQ5MVw04fw5p3OypX3f7o72CP\nU7l9J0WrgS5Rw7LCREQU05JmToPqn58CAEx//hDnr73e59lpREQXq9NgLS4uDnFx7o+1tLRApWr/\ny3tqaioMBgOMRiN0Op3wjE6ng8EgXbjCJSUlHnFxPf/LvV6v7fE7otHF9MuiO66BtfU8NGql233x\naprYjaMHIGNAsnCdIflUeOLvizT2izT2izT2C8W81lboM9KES0OdOYSNIaJo1eMCI06nVD1A3/fF\nTCZLT/946PVaGAxNPX5PtLnofnE6oe+XJFz+32OvoCxpJGRmK9Sq9oC61e4QqljNGH9pRPY7f1+k\nsV+ksV+kdbdfojHA457u2KTZWgLtkkcBAOeKlsGyfGWIW0RE0apbwVp8fDysVis0Gg3OnDmD9PR0\npKenw2g0Cs/U1dVhzJgxvdZQCgyb3eF2bhrQPjt4PYAxdoeQ3giAqY5ERCL+9nRPmzYNa9euRVlZ\nGfLy8rB+/XqUlZVBqVRi9uzZmDJlCpKTk/28ncKVPj1R+Nl4/CSc2kQ/TxMR9Uy3dr5OmDABe/bs\nAQB88MEHmDhxIkaPHo1Dhw7BbDbj3LlzqKysxLhx4zp5E4WKo60N2/dWuwVqlkQdTp/uKArjqlyl\nVircfiYioo493enp6cK98vJy3HLLLQDa93T/85//xMGDB4U93RqNRtjTTZFFUXNMCNTa9Okw1JkZ\nqBFRwHW6snb48GG89NJLqK2tRVxcHPbs2YNXX30Vy5cvR2lpKQYMGIC8vDwolUosWbIEhYWFkMlk\nWLhwoVBshELLJlohcwVb/fsnY5HoGdfetJx9NSjIyQxBK4mIIksg93QD3d/XHY3ppr2pW/1z883A\n3/7W/vOnn0I+YQKk6yhHNv7u+Ma+8Y/941tP+6bTYG3EiBHYunWr1/3Nmzd73cvNzUVubm6PGkTd\nIxWQOdraULqvBlXVBjSYbdAlqpGVqXcrLwy4FxGpqjZi1qQhXEEjIuqhnuzpBrq3r5v7LP276P7x\nVUQkCvuYvzu+sW/8Y//41tW+8RfQ9bjACIWWr4Ds4blZKN1Xg70VJ4Vnb9pXinuLS4RrqUqPpiYr\nGpttPLiTiKgbuKc7emi2bYG26BEAwLmipbAsXxXiFhFRLGKwFuE8A7J6sw17K05CpYpDVXVHms2u\ntXlu35tfvBsw27zel6LVCAVFiIjo4rj2dM+cOdNtT/fKlSthNpuhUChQWVmJFStWhLqp5IdbEZGa\nE3AmJvl5mogocBisRTCb3eEWkImVHzmNerMNSZaz2LZhvnD/7gc2wZSgw/UZSaj/qs7re1mZaUyB\nJCLqAu7pjj6K48egG38NAKAtLQ31X30T4hYRUaxjsBahbHYHvqltRIPE6hgANJitXqtp4rTH6hNn\nMSg9ARarHaYmm3B+Wn720IC2m4goWnBPd3RJuv1/oPr7xwAA0/99iPPXXR/iFhERMViLOOI9avVm\nG+QyQGqv+s41HYHa1ht/gt9d734Aa0NTKxqaWjE5awCmXncpz08jIqLY5KuICBFRGOjWOWsUOq49\navUXVtTaPAK1XWvz3FbUZhTtwJ9+lO/zfV8eb2CgRkREMUnz7m+EQO3c4scZqBFR2OHKWgTxt0dN\nLgP+tEY67bGPOg42ez1yd08AABsASURBVKvk91j9kYiIYhGLiBBRJGCwFkEam22Se9TWbV2MKwzf\nCteeJfkbz7UiOUGFs83eARurPxIRUSxRfFMD3Q1jAbCICBGFP6ZBRpCkBDV0ie6B1a61eX4DNQDQ\naTXIGpbmdR9g9UciIoohOTlCoGba9QEDNSIKe1xZiyBqpQJZmXrsrTiJUd8fxC/KnhY+W7ftM0Au\nB0Rnrrm4qjwqFHJUVRtharKy+iMREcUOFhEhogjFYC3C5GcPxaKCcW731m2vcAu6pAIyhVyOgpxM\nzJo0BI3NNhYVISKimKDZvhXaxQvbL1asgGHx8tA2iIjoIjBYC2M2u8M9sLLb0X9gqvD58T/sgfqG\n61AgCrpcAZlCpYSj1e4VkKmVChYTISKimOBWROTY90gbOggwNIWwRUREF4fBWhgSn6XWYLZBl6hG\nSfE0t2cMdWYk+vi+WqmAPq0vDByQiIgoBrkVEUlJQf3R70LcIiKi7mGBkTBjszuw+S//Fs5ScwJu\ngZr1tlnMtSciIvIhadaPO4qI7NzDQI2IIhpX1sKEeDXNdeC1+HBrAJhf/D6eu/96sNA+ERGRB7sd\netFWAU5sElE04MpamCjdVyOspgHegdqMoh1oMFvxTW0jbHZHKJpIREQUltS/3SYEapZFSxioEVHU\n4MpaCHgWDrHZHaiqNgAA8ip2oPDjEuFZ8blpMhnw6ntfQJeoRlamXqjySEREFKs8i4g4k5JD2Boi\not7FYC2IpAqHZGXqMTlrIBrMNsnVNLE2Z/v/1ptt2HvhPLWCnMygtJ2IiCicuBURSU5GffX3IW4R\nEVHvY7AWRK5URxdX0OX84QfsFAVqsx8phU3ZsTNNBsAp8b6qaiNmTRrC89KIiCimJM2eCdXH+wEA\nZ3e+D/sNE0LcIiKiwGCwFgQ2uwOGsy2oPFrn9Vlnq2ljM9NQVW2UfK+pyYrGZhvPTSMiotjgWUTk\nTGP7HgEioijFYC2APNMePVfHxIHaylmrcfCy0dCoFGi1O5Ci1SArMw15E6/A06fLhcIjYilaDZIS\nWBuSiIiin/q325C4aAEAwPLIYzi3anWIW0REFHgM1gLIM+3Rxd9qWrw6Divuugb65D5CemNWpl7y\nPVmZaUyBJCKiqMciIkQUqxisBYi4wqOYOFD7639Nxi9zF7l9frbZBlWc3C0Iy88eCqB9j5qpySqs\nurnuExERRSP5N8eRekMWAKAtMQn1NSdC3CIiouBisBYgjc02NIhSFws/2oS8yp3CtefeNBep1EaF\nXI6CnEzMmjTEreQ/ERFRtEqamwfVR/sAAGf/tBv28TeGtkFERCHQrWCtvLwcixYtwrBhwwAAmZmZ\nuO+++7Bs2TI4HA7o9Xq88sorUKlUvdrYSNJHHYekBBXONrd6pT3OL96NGzKSceCrM17f85faqFYq\nWEyEiIiiG4uIEBEJur2ydt111+H1118Xrp944gkUFBRg2rRpWLt2LcrKylBQUNArjQwXnodZSxEX\nFUn+z1Fs3fqY8JlrNS3nwoHWCfFKpjYSERFdoH7vXSQ++jMALCJCRAT0YhpkeXk5Vq9u/z/VyZMn\nY9OmTVETrPk6zDo/eygUcrnbs66iIlJFRFITOwIypjYSERF1cCsiUv0dnMkpIWwNEVF46HawVlNT\ng4ceegiNjY14+OGH0dLSIqQ9pqamwmDwLq4RqXwdZg0ABTmZwn2b3YFDX9W6BWoFP/sNmvokIiVB\njafmj4M23j01lKmNREQUy+T/+Qap148BALRpE1F/3Lv6MRFRrOpWsHb55Zfj4YcfxrRp03DixAn8\n9Kc/hcPhED53Oj1PFJOWkhKPuLierybp9doev8MXa+t5fHm8XvKzL4/X48FZfaBRxcHaeh56tRLv\niD4XFxFpPGdDn74a6NP6BqytngLZL5GM/SKN/SKN/SKN/UK9ISn/Nqj2/xUAi4gQEUnpVrDWr18/\nTJ8+HQBw6aWXIi0tDYcOHYLVaoVGo8GZM2eQnp7e6XtMJkt3/ng3er0WBkNTj9/jy0lDMwymFsnP\njGdbUP2NEfurarGoYJxwvzhvJT6/YpzbsylaDRyt9oC2VSzQ/RKp2C/S2C/S2C/SutsvDPCiW1f2\ndQtYRISIqEu6Fazt3LkTBoMBhYWFMBgMqK+vx+233449e/Zg5syZ+OCDDzBx4sTebmtQufapVR6t\ng691whStBnH33YtFn+4W7vkqyc8DrImIYkOsVUy+mH3dAKAu3Y7ERx4CAFgWLsK5p58NdpOJiCJG\nt4K17OxsPP744/jrX/8Ku92O4uJiDB8+HD//+c9RWlqKAQMGIC8vr/MXhTHPfWpSSopzhZ93jP0x\nfn3zvcK1XAY4AehY5ZGIKObEUsXkru7rBlhEhIjoYnUrWEtISMCGDRu87m/evLnHDQoHNrsDVdW+\nC6TccqoKi3/bUU5YajXN6QQenzcGVwxM4ooaEVGMi9aKyf7Gy6pqI2ZNGgK1UuFeRCRBi/pvaoPZ\nTCKiiNVrpfujSWOzDQ1mm+Rn4kqPTrkc9zz1Z0DiWV2ihoEaEVGM6o2Kyd0twhXMvYGnjOfQ0CQ9\nXpqarFColND/dA6w+8J2gY8+gnzSJOiD1kJv3DvpG/vGN/aNf+wf33raNwzWJCQlqKFLVKNeFIRp\nW8zY/uZPhWvDKROgUCBrb7VkuiT3qBERxabeqpjcnSJcwS6K47A7oNO6j5cuqfFKXKJPEK6FIiIh\nLNrDokG+sW98Y9/4x/7xrat94y+g8975S1ArFcjK7Jj327U2zz1QqzMDivZALD97KHLGZSA1UQO5\nDEhN1CBnXAb3qBERxShXxWSZTCZUTG5sbITVagWALldMjgSe46XL5K/2Y9Oz/w0AsCx4tH3cZLVH\nIqKL9v/bu/ewqOp1D+DfYYaL3GSQmQpMT9JRj4aIWwtFUxMvO8pM8hLR5Sm3FvKoeUyQSCvvSh4V\nu5hitSkTw0x229RNpeUOUcQHlXITWCYYwggMMMAwMOv8YUzizCDCwJrL9/OXM2sxvL6zmPd5Z631\n/nhmzYxZD90LCAIWPjXC8NyOtw/h0chQ3Hi+TOrkhKjw/ogcG9j+kcVERGS3HGFi8o1avpw8U6BC\nZU0DDrz15+0CHCJCRNQ5DtmstWctmJ6vvIyFqX8OTCkuqcS0NpowV2cplHJ3i8dKRHQ7Dh78By5e\nLEJs7CKxQ3FYjjAx+UYtX1o+EdwLAYPvAQAI7h5Q/fq7yJEREZlnK/XSoZq19q4Fc+No4cqMw2gK\nHQlXMQImIiKbY+8Tk01x3bcXipfmAACq9v8TujD7OXNIRCQmh2nWtLpmpB7+D344X2p47ua1YJx/\nOA6faQ8btpeXVXd7nEREnVVaegUxMXMQENAb586dxeOPR6KoqBA//ngejz8+A5GRM8UOkeyFXg/f\nB4ZCeulXNI4ZC3V6Bu9NIyKbYQv10u6btRvPppmaVgVcv85+YdRww+NNUxbhbOhkhGQWGJ11IyKy\nBT//XIC1a5NQXV2Np5+eic8+y0BjYyNefXWpVRQfsn3S/PPwHT8KAFCVth+68RNEjoiI6PZZe720\n+2Yt7ZtCk6P1W3jW1+DDG9ZOMyxwfdNZNyKijpA/+ABkF36y2Os1DfwfVH6Xfcv9AgJ6o2dPHzg7\nu0Au94VCoURdXR00mlqLxUKOyzNuMXp8sBMAUH7pKtCjh8gREZEtE6tWAtZfL+26WdPqmnGmwPzC\no7FH3sbk8/8CABwMnoJ3J7xotM+ZAhUixwZywiMRdUh7i4WlSaVSk/9u7xpfRKZIqirh178vAEAT\nn4i6xUtFjoiI7IFYtRKw/npp182aulaLChOXPkoEPTL+b7rh8WOL9kHvZLoZq6xpgLpWy0mPRETk\n0Fz37YX3H0NErp06C33f/xI3ICIiB2DXN2P19HSFr3frOY73lhYaGrVDQZPw6OIvzDZqACD3ckNP\nT86CJCIiB6XXQ/7AUHi/NAeNY8ai/KqajRoRUTeRCCKe4ysvr+n0aygUXm2+zu7MAsO9Z3FfbsDo\ngh8AACu3fo1Tv9bgVv/58OG9bfKetVvlxVExL6YxL6YxL6Z1NC8KhVcXRGO/OppjSx6zrYaI7Pkc\nuofCLfbaYuDftHnMjXnMTduYH/Pam5u26qPdXgbZsvD1tDH94P9TLmasnAsASP1rDCqefh7PjOmH\nopRssxMifb1cMWzA9TXYiIiIHA2HiBARic/umrWbF77O2DQNff/YVvCvExg/aIBhWEhIf4XJSZFh\n992J6MkDOFSEiIgcDoeIEBFZD7tr1lpG9Xs01CLjnWjD81t25yAquPXljC1nzc4UqFBZ0wC5lxtC\n+vtxbTUiInJIrp9/Bu8XXwDAISJERNbArpq1llH9E85/jUVHkgEAb057Faf6jUAvEyP4pU5OiArv\nj8ixgVDXatHT05Vn04iIyPHo9ZCPHAbZLxfRGDYG6s+/BCQSsaMiInJ4dtWsVVfUYNvaSHhq65Db\ndyhWTF9hKDZtjeB3dZZyND8RETmk1kNE9kH30ESRIyIiohZ206w5f3cUg56YCgCIm7UGPwYMarWd\nI/iJiIha81y2BD1S3gfAISJERNbI9m/MEgT4PBwOnyemovkuf2xNPWHUqAFASH8/XuJIRPSHwsKf\n8dtvl9q177VrKmzYsLqLI6LuJFFXQaH0Ro+U96GJT0R5WTUbNSKim9xOrbzRt99mWiwG227W9HrA\nyQnOOSdRvX0XKvIuYObEgQgf3hu9vN3gJAF6ebshfHhvjuAnIqul1TWjrLIOWl1zt/3OY8e+weXL\nv7Vr3169/LB06atdHBFZmrnjynV/Ovz+uw+A60NEOO2RiGyBtdfKFjqdDmlpuy0Wg01fBqltbELN\n4qWonhcLF7kPAA4NISLbcfNSI77ergjpr+jURNq//e1ZvP76agQE9EZZ2VXEx/8vdu36uNU+RUWF\nOHDgcxw79g3kcjnefPM1hIaGQS6XY9SoMdi0aT1kMhmcnJywcuU6aDQaJCbGISUlFbNmTUNU1JM4\nciQTjY2N2LLlHbi7e1giHWQhZo+rcf3gN3oEZBeLOESEiGyGtdRKnU6H7dvfhkwmg1J5B+LiEqHV\narF8eTwaGxuh0+mweHEcvvzyAIqKCpGUtA7r13f+qhSbPLPWrNdjd2YBEnedwlOSUXj103zszixA\ns15v2KdlaAgbNSKyVi1LjVyr1kIAcK1ai8ycYqR9U9jh15wy5WF8/fURAMDx498hPHyy0T6Bgffi\ngQdGYt68WAwadB+ampoQGjoKzz77AqqqKvDyy68gOXk7goKCceTIV61+trm5Gf369cPbb++Av78/\ncnJOdThW6hqmjqufvzqOO/19IbtYhKo9+6De/082akRkE6ylVm7evBHr1r2FrVvfg6+vL779NhOn\nT5+EQqHEtm3vY/nylaisrEBU1NPo06cvliyJ73B8N7LJZq3VmyZY5k0jIupOLUuNmHKmQNXhyzzC\nwyfj2LFvAQA//PA9Jk40LkCmDBo0GAAgl/fC9u3vIDZ2LjIzD0OtVhvtO3z4cACAQnEHNJraDsVJ\nXaOhscnouJr7zfvYlroIAFBSWMJpj0RkM6ylVlZUXENx8WUkJLyC2Ni5yM09jfLyMgwePAT5+eew\nceMalJQUIzR0VIfiaYvFL4Ncs2YN8vLyIJFIkJCQgCFDhlj09W/1pt28lhoRkTVS12pRUa01ua2t\npUZupWdPHyiVSvz0Uz70egEKhbJdPyeTOQMAtmxJwlNPPYvQ0FHYvTsV9fV1RvtKpVIA169kEATh\ntmOkrlNZ/edx5dFQiz3vRAMAPhn5JPaOmoU1zVK074ggIhKftdRKmcwZfn4KbNv2vtG2Dz/8FLm5\nOdi/Px35+ecwZUrEbcfTFoueWTt58iQuXbqEtLQ0rF69GqtXW356WHveNCIia9fT0xW+3qaXE+ns\nUiOTJz+MTZvWY/z4CWb3kUgkaG42/kZSra5CQEBvNDY24sSJf6OpqanDcVD3k3v/eVz99ewhAMCc\n59/DnpGzuIQNEdkca6mV3t7eAIBffrkIAEhP34PCwp9x6lQ2Tp3Kxv33h+Lll1/BhQs/QiJxMllf\nO8qizVpWVhbCw8MBAIGBgVCr1aittewlMl35phERdRdXZylC+itMbuvsUiNhYQ+iuLgY48aZL0DB\nwSHYvHkjcnJOtno+MnIWli1bgtdei0Nk5Cx89dWXFv8cp67j5iIzHFfp9z+BRxd/gas+dwLgEjZE\nZHusqVbGxy/HmjVvICZmDs6ezUOfPn3Ru/fd+PvfdyE2di5WrVqBqKhn4Ofnh6YmHRIT4zoc240s\nehmkSqXC4MGDDY99fX1RXl4OT09Pi/2OljctM6fYaBsLERHZkpYlRc4UqFBZ0wC5lxtC+vt1eqmR\nc+fyEBY2Bl5eXmb3iYiYioiIqQCA9PR/GJ5/7LHpeOyx6YbHY8eOBwCkpKQa9vXw8EBdXQ1iYxd1\nKk7qGl11XBERicFaaiUA7NjxUavtd93lj3ffTTH6uY8//qxTsd2oS0f33+peBrncHTLZ7TdXsTND\n4N7DBSfO/w5VVT38fHog9L678PyjgyGV2uTMlC6hUJg/+BwZ82Ia82JaV+dl4ZN/QUNjEyqrtZB7\nu8LNpXMfy1u3bsXx48eRnJwMna4GcXHG3+yNGDECCxYs6NTv4fFivbiEDRHZk674TEtJ2Y7s7Cys\nXr0BpaWlWLVqudE+ISF/wQsvzOvU77EEiWDBu8OTk5OhUCgwe/ZsAMCECRNw4MABs2fWystrOvX7\ntLpmSF2c0dyoYyG6iULh1en82iPmxTTmxTTmxbSO5oUN3u3paI55zJrH/JjH3JjH3LSN+TGvvblp\nqz5a9DRUWFgYDh8+DADIz8+HUqm06CWQN3N1luIuPw82akREREREZHcsehnksGHDMHjwYMyePRsS\niQQrVqyw5MsTERERERE5DIvfs7ZkyRJLvyQREZHN6+p1SImIyP506YARIiIiar0OaVFRERISEpCW\nliZ2WEREZOU4OpGIiKiLdcc6pEREZH94Zo2IiKiLdWQd0o4ub8Opm21jfsxjbsxjbtrG/JjX2dyw\nWSMiIupm7Vk1p7Ky7rZflyO028b8mMfcmMfctI35Mc/qRvcTERGRMaVSCZVKZXhcVlYGhUIhYkRE\nRGQLLLooNhERERnLzc1FcnIyPvjgA+Tn52PVqlX49NNPxQ6LiIisHC+DJCIi6mJch5SIiDqCZ9aI\niIiIiIisEO9ZIyIiIiIiskJs1oiIiIiIiKwQmzUiIiIiIiIrxGaNiIiIiIjICtn0NMg1a9YgLy8P\nEokECQkJGDJkiNghiWbDhg04ffo0mpqaMG/ePAQFBWHp0qVobm6GQqHAxo0b4eLiInaYomhoaMAj\njzyCmJgYjBw5knkBkJGRgZ07d0Imk2HBggUYMGCAw+dFo9EgLi4OarUaOp0O8+fPh0KhwOuvvw4A\nGDBgAN544w1xg+xGBQUFiImJwXPPPYfo6Gj8/vvvJo+RjIwMfPTRR3BycsLMmTMxY8YMsUN3WKyJ\nxlgb28b6aBprpGmsk6Z1eb0UbFR2drYwd+5cQRAEobCwUJg5c6bIEYknKytLmDNnjiAIglBRUSGM\nHTtWiI+PFw4ePCgIgiC89dZbwieffCJmiKLatGmTMH36dGHfvn3Mi3D9GJk0aZJQU1MjXL16VUhM\nTGReBEFITU0VkpKSBEEQhNLSUmHy5MlCdHS0kJeXJwiCICxevFg4evSomCF2G41GI0RHRwuJiYlC\namqqIAiCyWNEo9EIkyZNEqqrq4X6+nohIiJCqKysFDN0h8WaaIy18dZYH42xRprHOmmsO+qlzV4G\nmZWVhfDwcABAYGAg1Go1amtrRY5KHCNGjMCWLVsAAN7e3qivr0d2djYmTJgAABg/fjyysrLEDFE0\nRUVFKCwsxLhx4wCAecH1v52RI0fC09MTSqUSK1euZF4AyOVyVFVVAQCqq6vh4+ODkpISw9kJR8qL\ni4sLduzYAaVSaXjO1DGSl5eHoKAgeHl5wc3NDcOGDUNubq5YYTs01kRjrI1tY300jTXSPNZJY91R\nL222WVOpVJDL5YbHvr6+KC8vFzEi8UilUri7uwMA0tPT8eCDD6K+vt5wir5Xr14Om5v169cjPj7e\n8Jh5AYqLi9HQ0IAXX3wRUVFRyMrKYl4ARERE4MqVK5g4cSKio6OxdOlSeHt7G7Y7Ul5kMhnc3Nxa\nPWfqGFGpVPD19TXs48ifw2JjTTTG2tg21kfTWCPNY5001h310qbvWbuRwLW9kZmZifT0dOzatQuT\nJk0yPO+oufniiy8wdOhQ3H333Sa3O2peAKCqqgrbtm3DlStX8Mwzz7TKhaPm5cCBA/D390dKSgou\nXLiA+fPnw8vLy7DdUfNiirlcMEfWg+/Fn1gbjbE+to010jTWydtniXpps82aUqmESqUyPC4rK4NC\noRAxInF9//33eO+997Bz5054eXnB3d0dDQ0NcHNzw9WrV1udnnUUR48exeXLl3H06FGUlpbCxcWF\necH1b3lCQkIgk8nQp08feHh4QCqVOnxecnNzMXr0aADAwIEDodVq0dTUZNjuqHlpYepvx9Tn8NCh\nQ0WM0nGxJprG2mga66N5rJHmsU62j6Xrpc1eBhkWFobDhw8DAPLz86FUKuHp6SlyVOKoqanBhg0b\nsH37dvj4+AAARo0aZcjPkSNHMGbMGDFDFMXmzZuxb98+7N27FzNmzEBMTAzzAmD06NE4ceIE9Ho9\nKisrUVdXx7wA6Nu3L/Ly8gAAJSUl8PDwQGBgIHJycgA4bl5amDpGgoODce7cOVRXV0Oj0SA3NxfD\nhw8XOVLHxJpojLXRPNZH81gjzWOdbB9L10uJYMPnLJOSkpCTkwOJRIIVK1Zg4MCBYockirS0NCQn\nJ+Oee+4xPLdu3TokJiZCq9XC398fa9euhbOzs4hRiis5ORkBAQEYPXo04uLiHD4ve/bsQXp6OgDg\npZdeQlBQkMPnRaPRICEhAdeuXUNTUxMWLlwIhUKB5cuXQ6/XIzg4GMuWLRM7zG5x/vx5rF+/HiUl\nJZDJZLjjjjuQlJSE+Ph4o2Pk0KFDSElJgUQiQXR0NKZOnSp2+A6LNbE11sb2YX00xhppGuukse6o\nlzbdrBEREREREdkrm70MkoiIiIiIyJ6xWSMiIiIiIrJCbNaIiIiIiIisEJs1IiIiIiIiK8RmjYiI\niIiIyAqxWSMiIiIiIrJCbNaIiIiIiIisEJs1IiIiIiIiK/T/QIAVW7wfn2EAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f1e14d82fd0>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "xAP1EoQi86XB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Inference"
      ]
    },
    {
      "metadata": {
        "id": "K2yfNk3d8-Vj",
        "colab_type": "code",
        "outputId": "ac0cca73-6124-435d-e3cc-49b373d5e64d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 159
        }
      },
      "cell_type": "code",
      "source": [
        "# Feed in your own inputs\n",
        "X_infer = np.array((0, 1, 2), dtype=np.float32)\n",
        "standardized_X_infer = X_scaler.transform(X_infer.reshape(-1, 1))\n",
        "pred_infer = (lm.predict(standardized_X_infer) * np.sqrt(y_scaler.var_)) + y_scaler.mean_\n",
        "print (pred_infer)\n",
        "df.head(3)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[10.50158072 14.14515288 17.78872504]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>X</th>\n",
              "      <th>y</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.0</td>\n",
              "      <td>3.830389</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.0</td>\n",
              "      <td>16.092175</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.0</td>\n",
              "      <td>16.054555</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     X          y\n",
              "0  0.0   3.830389\n",
              "1  1.0  16.092175\n",
              "2  2.0  16.054555"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "metadata": {
        "id": "PHH0fYp_BYC5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Interpretability"
      ]
    },
    {
      "metadata": {
        "id": "OhXo8CbPBZ-G",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Linear regression offers the great advantage of being highly interpretable. Each feature has a coefficient which signifies it's importance/impact on the output variable y. We can interpret our coefficient as follows: by increasing X by 1 unit, we increase y by $W$ (~3.65) units. \n",
        "\n",
        "**Note**: Since we standardized our inputs and outputs for gradient descent, we need to apply an operation to our coefficients and intercept to interpret them. See proof below."
      ]
    },
    {
      "metadata": {
        "id": "JZxnrDuCBbK9",
        "colab_type": "code",
        "outputId": "2a207e46-5939-4704-cf20-747e743459e1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Unstandardize coefficients \n",
        "coef = lm.coef_ * (y_scaler.scale_/X_scaler.scale_)\n",
        "intercept = lm.intercept_ * y_scaler.scale_ + y_scaler.mean_ - np.sum(coef*X_scaler.mean_)\n",
        "print (coef) # ~3.65\n",
        "print (intercept) # ~10"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[3.64357519]\n",
            "[10.50158285]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "yVmIP13u9s33",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Proof for unstandardizing coefficients:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "ViDPSLbR9v4B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note that both X and y were standardized.\n",
        "\n",
        "$\\frac{\\hat{y} - \\mathbb{E}[y]}{\\sigma_y} = W_0 + \\sum_{j=1}^{k}W_jz_j$\n",
        "\n",
        "$z_j = \\frac{x_j - \\bar{x}_j}{\\sigma_j}$\n",
        "\n",
        "$ \\hat{y}_{scaled} = \\frac{\\hat{y}_{unscaled} - \\bar{y}}{\\sigma_y} = \\hat{W_0} + \\sum_{j=1}^{k} \\hat{W}_j (\\frac{x_j - \\bar{x}_j}{\\sigma_j}) $\n",
        "\n",
        "$\\hat{y}_{unscaled} = \\hat{W}_0\\sigma_y + \\bar{y} - \\sum_{j=1}^{k} \\hat{W}_j(\\frac{\\sigma_y}{\\sigma_j})\\bar{x}_j + \\sum_{j=1}^{k}\\hat{W}_j(\\frac{\\sigma_y}{\\sigma_j})x_j $\n"
      ]
    },
    {
      "metadata": {
        "id": "rToCXKqeJcvj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Regularization"
      ]
    },
    {
      "metadata": {
        "id": "L4GFv8xRJmOZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Regularization helps decrease overfitting. Below is L2 regularization (ridge regression). There are many forms of regularization but they all work to reduce overfitting in our models. With L2 regularization, we are penalizing the weights with large magnitudes by decaying them. Having certain weights with high magnitudes will lead to preferential bias with the inputs and we want the model to work with all the inputs and not just a select few. There are also other types of regularization like L1 (lasso regression) which is useful for creating sparse models where some feature cofficients are zeroed out, or elastic which combines L1 and L2 penalties. \n",
        "\n",
        "**Note**: Regularization is not just for linear regression. You can use it to regularize any model's weights including the ones we will look at in future lessons."
      ]
    },
    {
      "metadata": {
        "id": "D_OcpRxF-Oj7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* $ J(\\theta) = = \\frac{1}{2}\\sum_{i}(X_iW - y_i)^2 + \\frac{\\lambda}{2}W^TW$\n",
        "* $ \\frac{\\partial{J}}{\\partial{W}}  = X (\\hat{y} - y) + \\lambda W $\n",
        "* $W = W- \\alpha\\frac{\\partial{J}}{\\partial{W}}$\n",
        "where:\n",
        "  * $\\lambda$ is the regularzation coefficient"
      ]
    },
    {
      "metadata": {
        "id": "HHaazL9f8QZX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Initialize the model with L2 regularization\n",
        "lm = SGDRegressor(loss=\"squared_loss\", penalty='l2', alpha=1e-2, \n",
        "                  max_iter=args.num_epochs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VTIUZLbGZP4e",
        "colab_type": "code",
        "outputId": "f6ea31dd-889a-4d49-9dfe-71ce99734dae",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        }
      },
      "cell_type": "code",
      "source": [
        "# Train\n",
        "lm.fit(X=standardized_X_train, y=standardized_y_train)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/stochastic_gradient.py:183: FutureWarning: max_iter and tol parameters have been added in SGDRegressor in 0.19. If max_iter is set but tol is left unset, the default value for tol in 0.19 and 0.20 will be None (which is equivalent to -infinity, so it has no effect) but will change in 0.21 to 1e-3. Specify tol to silence this warning.\n",
            "  FutureWarning)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SGDRegressor(alpha=0.01, average=False, early_stopping=False, epsilon=0.1,\n",
              "       eta0=0.01, fit_intercept=True, l1_ratio=0.15,\n",
              "       learning_rate='invscaling', loss='squared_loss', max_iter=100,\n",
              "       n_iter=None, n_iter_no_change=5, penalty='l2', power_t=0.25,\n",
              "       random_state=None, shuffle=True, tol=None, validation_fraction=0.1,\n",
              "       verbose=0, warm_start=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "metadata": {
        "id": "ORwkUqcuZhbX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Predictions (unstandardize them)\n",
        "pred_train = (lm.predict(standardized_X_train) * np.sqrt(y_scaler.var_)) + y_scaler.mean_\n",
        "pred_test = (lm.predict(standardized_X_test) * np.sqrt(y_scaler.var_)) + y_scaler.mean_"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "IWCvYxBxZhd5",
        "colab_type": "code",
        "outputId": "be12b93e-f7a2-45b1-e0f1-bd74fe4d7288",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Train and test MSE\n",
        "train_mse = np.mean((y_train - pred_train) ** 2)\n",
        "test_mse = np.mean((y_test - pred_test) ** 2)\n",
        "print (\"train_MSE: {0:.2f}, test_MSE: {1:.2f}\".format(\n",
        "    train_mse, test_mse))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train_MSE: 30.22, test_MSE: 35.73\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "mdNX2W5eh2ma",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Regularization didn't help much with this specific example because our data is generated from a perfect linear equation but for realistic data, regularization can help our model generalize well."
      ]
    },
    {
      "metadata": {
        "id": "C2mrVS4UZp3Q",
        "colab_type": "code",
        "outputId": "9cecf0b3-2c37-4a29-f9f9-fa139ce0863d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Unstandardize coefficients \n",
        "coef = lm.coef_ * (y_scaler.scale_/X_scaler.scale_)\n",
        "intercept = lm.intercept_ * y_scaler.scale_ + y_scaler.mean_ - (coef*X_scaler.mean_)\n",
        "print (coef) # ~3.65\n",
        "print (intercept) # ~10"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[3.60760035]\n",
            "[12.17506412]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "V74lNFE5v5pQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Categorical variables"
      ]
    },
    {
      "metadata": {
        "id": "2r6Xhyg7v5vX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In our example, the feature was a continuous variable but what if we also have features that are categorical? One option is to treat the categorical variables as one-hot encoded variables. This is very easy to do with Pandas and once you create the dummy variables, you can use the same steps as above to train your linear model."
      ]
    },
    {
      "metadata": {
        "id": "unhcIOfMxQEQ",
        "colab_type": "code",
        "outputId": "a359bedd-0b16-4ca7-9b41-ef0ea1623380",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        }
      },
      "cell_type": "code",
      "source": [
        "# Create data with categorical features\n",
        "cat_data = pd.DataFrame(['a', 'b', 'c', 'a'], columns=['favorite_letter'])\n",
        "cat_data.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>favorite_letter</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>a</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>b</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>c</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>a</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  favorite_letter\n",
              "0               a\n",
              "1               b\n",
              "2               c\n",
              "3               a"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "metadata": {
        "id": "m4eQmJdrxQGr",
        "colab_type": "code",
        "outputId": "64a09a9b-3d8b-47d5-e1bc-edb795291e6c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        }
      },
      "cell_type": "code",
      "source": [
        "dummy_cat_data = pd.get_dummies(cat_data)\n",
        "dummy_cat_data.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>favorite_letter_a</th>\n",
              "      <th>favorite_letter_b</th>\n",
              "      <th>favorite_letter_c</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   favorite_letter_a  favorite_letter_b  favorite_letter_c\n",
              "0                  1                  0                  0\n",
              "1                  0                  1                  0\n",
              "2                  0                  0                  1\n",
              "3                  1                  0                  0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "metadata": {
        "id": "B5R8x-KyiBWJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now you can concat this with your continuous features and train the linear model."
      ]
    },
    {
      "metadata": {
        "id": "eVOXoCRsokzp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# TODO"
      ]
    },
    {
      "metadata": {
        "id": "4c7ttuUwfeLA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "- polynomial regression\n",
        "- simple example with normal equation method (sklearn.linear_model.LinearRegression) with pros and cons vs. SGD linear regression"
      ]
    }
  ]
}
